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organizations,  which  many  have  addressed  by  contracting  out 
some  of  the  work.  The  Air  Force  Deputy  Assistant  Secretary 
(Contracting),  Mr.  Charlie  Williams,  sponsored  a  study  to 
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have  the  authors  assess  the  current  status  of  contracting  out 
procurement  functions  within  DoD  and  federal  agencies. 
This  study  determined  that  government  agencies  display 
considerable  variety  in  their  use  of  contractor  support 
for  procurement  functions.  This  article  summarizes  the 
current  status  of  contracting  out  procurement  functions 
and  recommends  that  contracting  managers  retain  a  limited 
capability  to  contract  out  to  meet  their  mission  requirements. 
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government’s  preferred  approach  for  service  contracting,  but 
despite  great  efforts  and  training,  PBS  A  remains  difficult  to 
implement.  An  analysis  of  services  acquisition  suggests  that 
while  PBSA  may  be  useful  for  routine,  common,  and  relatively 
simple  services,  it  is  not  as  applicable  for  services  that  are 
too  long-term  and  complex  to  permit  complete  specification 
of  results  and  competitive  pricing  at  the  outset  of  contracting. 

A  new  approach  for  contracting  these  kinds  of  services  is 
recommended. 
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Current  business  scholars  consider  customer  focus  critical  to 
competitive  advantage.  The  literature  is  full  of  research  and 
recommendations  considering  the  what  and  how  of  customer 
focus.  Modern  defense  product  developers,  like  all  modern 
business  enterprise,  seek  competitive  advantage.  Customer 
focus,  and  the  promise  of  competitive  advantage  within  that 
concept,  is  seen  as  a  critical  component  of  a  modern  defense 
company’s  strategy.  This  article  explores  the  difficulty  of 
developing  true  customer  focus  within  the  rather  strict  and 
regulated  Army  procurement  system. 
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LESSONS  LEARNED  IN  ACQUISITION  MANAGEMENT 

Dennis  K.  Van  Cemert  and  Martin  Wartenberg 

Many  projects  and  programs  fall  short  of  meeting  their  initial 
intended  goals.  Tracing  these  shortfalls  to  their  common  set  of 
root  causes  and  analyzing  these  root  causes  to  find  common 
threads  illustrates  opportunities  for  lessons  learned.  The  authors 
of  this  article  examined  these  common  threads  and  referred 
to  their  professional  experience  in  defense  acquisition  and 
academic  backgrounds  in  project  management  and  systems 
engineering  to  address  these  issues  and  propose  strategies  for 
countering  their  ill  effects  on  program  performance. 


TEST  AND  EVALUATION  LESSONS  LEARNED 
FROM  THE  FIELD 

Karen  M.  Stadier 

This  article  examines  test  and  evaluation  (T&E)  lessons 
learned  from  more  than  300  students  with  extensive  T&E  field 
experience  who  attended  the  Defense  Acquisition  University 
(DAU)  test  and  evaluation  classes  during  FY02-FY05.  The 
T&E  lessons  learned  in  18  categories  were  researched  and 
correlated  and  findings  in  the  top  five  categories  are  presented. 
In  particular,  this  article  focuses  on  detailed  lessons  learned  in 
the  areas  of  test  design  and  execution,  test  planning,  teamwork 
and  communication,  funding,  and  scheduling.  A  compilation 
of  student  (field  practitioner)  comments  and  recommendations 
is  presented,  and  overall  results  are  compared  with  results  from 
other  similar  studies  and  documents. 
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Welcome  to  the  Defense  Acquisition  Review  Journal  (ARJ)  theme  edition 
on  contracting  trends  in  acquisition.  Our  featured  author  for  this  edition  is 
Professor  John  Krieger,  the  Director  of  the  Contracting  Center  of  the  Defense 
Acquisition  University’s  Curriculum  Development  Support  Center.  In  his  article, 
“Professionalism  in  the  Acquisition  Contracting  Workforce:  Have  we  gone  too 
far?,”  Professor  Krieger  questions  the  basic  required  credentials  of  the  acquisition 
contracting  workforce.  The  Defense  Acquisition  Workforce  Improvement  Act 
(DAWIA)  threshold  requirements  for  this  career  field  include  a  bachelor’s  degree 
and  24  semester  hours  of  business-related  courses.  Since  recruiting  and  retaining 
acquisition  professionals  in  this  career  field  have  become  increasingly  difficult,  it 
may  be  time  to  reconsider  the  basic  eligibility  criteria  for  hiring.  The  author  asserts 
that  experience  may  be  equal  in  importance  to  education,  and  that  a  wider  variety  of 
educational  and  professional  backgrounds  for  members  of  the  acquisition  contracting 
workforce  would  expand  the  pool  of  talented  candidates.  He  argues  that  we  should 
hire  the  best  and  brightest,  no  matter  their  degree  areas,  and  then  train  them  to  the 
specific  knowledge,  skills,  and  abilities  that  they  will  need  to  do  federal  government 
contracting,  which  is  not  what  they  would  have  learned  in  most  business  degree 
programs. 

The  following  article,  “Contracting  Out  Procurement  Functions:  Current  Status” 
by  J.  Scott  Williams,  Roland  D.  Kankey,  Billy  R.  Harry,  and  Alan  S.  Gilbreth, 
summarizes  a  large  research  study  sponsored  by  the  Air  Force  Deputy  Assistant 
Secretary  (Contracting)  to  assess  the  current  status  of  contracting  out  procurement 
functions  within  the  Department  of  Defense  (DoD)  and  federal  agencies.  For 
years,  the  DoD  acquisition  workforce  has  been  decreasing,  but  the  workload  has 
not.  This  has  created  a  dilemma  for  DoD  procurement  organizations,  causing 
these  organizations  to  contract  out  some  of  the  work.  The  study  determined 
that  government  agencies  display  considerable  variety  in  their  use  of  contractor 
support  for  procurement  functions,  and  the  article  summarizes  the  current  status  of 
contracting  out  procurement  functions  and  recommends  that  contracting  managers 
retain  a  limited  capability  to  contract  out  to  meet  their  mission  requirements. 
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The  third  article,  “A  Proposal  for  a  New  Approach  to  Performance-Based  Services 
Acquisition,”  by  Vernon  J.  Edwards  and  Ralph  C.  Nash,  Jr.,  discusses  the  concept  of 
Performance-Based  Services  Acquisition  (PBS  A).  Since  1991,  the  Office  of  Federal 
Procurement  Policy’s  specifies  that  PBSA  is  the  government’s  preferred  approach 
for  service  contracting.  Flowever,  despite  numerous  efforts — publication  of  many 
guidebooks  and  significant  investments  in  training  and  consultant  services — PBSA 
remains  difficult  to  implement.  An  analysis  of  services  acquisition  suggests  that 
while  PBSA  may  be  useful  for  routine,  common,  and  relatively  simple  services,  it  is 
not  applicable  for  services  that  are  too  long-term  and  complex  to  permit  complete 
specification  of  results  and  competitive  pricing  at  the  outset  of  contracting.  A  new 
approach  for  contracting  these  kinds  of  services  is  recommended. 

The  fourth  article,  “Customer  Focus  and  Army  Procurement:  Is  it  Possible?,”  by 
Keith  R.  Shelton  and  Dr.  Drumm  McNaughton,  examines  the  concept  of  customer 
focus  from  a  DoD  contractor’s  perspective.  Current  business  scholars  consider 
customer  focus  to  be  a  critical  factor  in  maintaining  competitive  advantage.  The 
literature  is  full  of  research  and  recommendations  considering  the  what  and  how 
of  customer  focus.  Modem  defense  product  developers,  like  all  modem  business 
enterprise,  seek  competitive  advantage.  Customer  focus,  and  the  promise  of 
competitive  advantage  within  that  concept,  is  seen  as  a  critical  component  of  a 
modern  defense  company’s  strategy.  This  article  explores  the  difficulty  of  true 
customer  focus  within  the  rather  strict  and  regulated  Army  procurement  system. 
Several  common  problems  are  discussed,  such  as  defining  the  customer,  the  inherent 
rigidity  of  the  procurement  process,  public  relations,  and  product  focus.  The  authors 
argue  that  by  understanding  the  customer  better,  a  contractor  can  create  the  necessary 
visions,  strategies,  and  trust  leading  to  a  successful  program.  Finally,  constant  focus 
on  the  end  user — the  combat  soldier — can  build  morale  and  enthusiasm  within  the 
firm  and  a  positive  brand  name  outside  the  firm. 

In  the  next  article,  “Lessons  Learned  in  Acquisition  Management,”  Dennis  K. 

Van  Gemert  and  Martin  Wartenberg  analyze  why  many  projects  and  programs  fail  to 
meet  their  initial  intended  goals.  Managing  project  scope  is  essential  to  meeting  all 
objectives.  Changing  scope  (such  as  requirements  creep  or  funding  cuts)  will  almost 
certainly  derail  any  original  estimates  of  program  performance.  Several  other  lessons 
learned  discussed  in  the  article  deal  with  the  following  areas:  immature  technologies, 
use  of  management  reserves,  risk  analyses,  effective  communications,  staffing  and 
resource  issues,  frequent  personnel  turnover,  overly  optimistic  contractor  claims, 
and  integrating  sound  systems  engineering  principles  into  program  management 
decisions.  The  authors  conclude  that  we  have  become  very  good  at  documenting 
lessons  learned,  but  not  so  disciplined  in  the  institutionalization  of  these  lessons. 
Documenting  lessons  learned  is  just  the  beginning  of  knowledge  management.  These 
lessons  must  be  socialized  among  program  participants  to  the  degree  that  they  are 
transferred  to  upcoming  generations. 

The  last  article,  “Test  and  Evaluation  Lessons  Learned  from  the  Field,”  by  Karen 
M.  Stadler,  summarizes  lessons  learned  reported  by  Defense  Acquisition  University 
(DAU)  students  taking  Advanced  Test  and  Evaluation  (TST-301)  over  a  4-year 
time-frame  (FY02-FY05).  As  part  of  TST-301,  all  students  prepare  and  present 
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briefings  on  test  and  evaluation  (T&E)  issues  and  lessons  learned  based  on  their 
actual  T&E  experiences  in  acquisition.  These  students  typically  have  many  years  of 
T&E/acquisition  field  experience,  and  their  presentations  contain  a  wealth  of  valuable 
information,  which  could  help  others  avoid  common  sources  of  error  when  designing 
and  executing  test  events.  Data  from  this  article  is  taken  from  a  sample  of  393 
TST-301  graduates,  and  lessons  learned  are  grouped  into  18  categories.  The  top  five 
categories  of  lessons  learned  are  Test  Design  and  Execution,  Test  Planning,  Teamwork 
and  Communication,  Funding/Budget/Cost,  and  Scheduling. 

Contracting  trends  in  acquisition  and  other  Defense  ARJ  themes  provide 
opportunities  for  dialog  among  members  of  the  Acquisition,  Technology,  and 
Logistics  (AT&L)  community.  Journal  readers  are  encouraged  to  share  their 
experiences  in  the  field,  any  materials  and  methodologies  that  verify  research 
conclusions,  tutorials,  and  fresh  viewpoints  regarding  subject  areas  relevant  to  the 
AT&L  workforce  by  sending  submissions  to  DefenseARJ@dau.mil. 


Dr.  Paul  Alfieri 
Executive  Editor 
Defense  ARJ 
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PROFESSIONALISM  IN  THE  ACQUISITION  CONTRACTING  WORKFORCE 


PROFESSIONALISM 
IN  THE  ACQUISITION 
CONTRACTING 
WORKFORCE: 

HAVE  WE  GONE  TOO  FAR? 


John  Krieger 


To  professionalize  the  acquisition  contracting  workforce,  the  Department  of 
Defense  (DoD)  currently  requires  a  bachelor's  degree  and  24  hours  of  business- 
related  courses  as  a  threshold  requirement  for  the  contracting  occupational 
series.  In  recent  years,  recruiting  and  retaining  needed  personnel  has  become 
increasingly  problematic.  It  may  be  time  to  reconsider  the  hiring  criteria  used  to 
select  candidates  for  the  acquisition  workforce.  Considering  experience  to  be  of 
equal  importance  to  education,  as  is  done  in  the  private  sector,  and  considering 
candidates  with  a  wider  variety  of  educational  and  professional  backgrounds  may 
enhance  the  pool  of  talented  candidates  available  for  the  acquisition  contracting 
field. 
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etween  15  and  20  years  ago,  the  Department  of  Defense  (DoD),  with  the 
assistance  of  legislation,  continued  its  efforts  to  professionalize  the  acquisition 
contracting  workforce  by  requiring  a  bachelor’s  degree  and  24  hours  of 
business-related  courses  as  a  threshold  requirement  for  the  contracting  occupational 
series.  Over  the  last  five  years,  there  has  been  a  bombardment  of  statistics  concerning 
the  “graying”  of  the  acquisition  workforce  and  the  number  of  civil  servants  eligible 
for  retirement. 

Although  the  tidal  wave  of  retirements  that  were  forewarned  in  some  of  the  more 
dire  predictions  have  not  appeared,  many  agencies  are  feeling  the  stresses  associated 
with  trying  to  recruit  needed  personnel,  including  positions  going  vacant  for  long 
periods  of  time  and  the  need  to  contract  for  advisory  and  assistance  services  to  meet 
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contracting  requirements.  These  stresses  may  be  due  to  decisions  that  were  made 
long  ago  in  an  effort  to  professionalize  the  acquisition  contracting  workforce  and  to 
stem  the  practice  in  some  organizations  of  selecting  insufficiently  qualified  personnel 
for  acquisition  contracting  positions  (i.e.,  GS-1102  contract  specialists). 

Although  the  motives  may  have  been  well-intentioned,  we  are  now  living  with 
the  result  of  unintended  adverse  consequences.  It  is  now  time  to  reassess  these  past 
decisions,  specifically  degree  and  course  requirements,  and  determine  whether  a 
mistake  in  approach  has  been  made.  By  selecting  the  wrong  solution  to  solve  a 
problem,  has  a  more  significant  problem  been  created?  The  BLUF  (bottom-line-up¬ 
front):  I  believe  we  have  made  a  mistake. 

My  opinion  is  somewhat  biased  on  this  question.  My  father,  my  two  brothers,  and 
I  have  well  over  a  century  of  combined  federal  government  service.  My  two  brothers 
and  I  have  all  been  contracting  officers  for  the  federal  government.  One  brother, 
while  a  member  of  the  Senior  Executive  Service  (SES),  was  the  director  of  contracts 
for  what  is  now  called  the  National  Reconnaissance  Office  (NRO).  I  am  currently 
the  director  of  the  Contracting  Center  of  the  Defense  Acquisition  University’s 
Curriculum  Development  Support  Center.  If  my  brothers  and  I  were  to  graduate 
from  college  today  with  the  same  bachelor’s  degrees  we  obtained  in  the  1960s  and 
1970s,  none  of  us  could  come  to  work  for  DoD  in  contracting.  The  ways  we  entered 
the  contracting  field  (one  through  his  secondary  occupational  code  in  the  U.S.  Air 
Force,  one  from  the  PACE  [Professional  and  Administrative  Career  Examination], 
and  one  from  his  placement  in  the  top  ten  percent  of  his  class,  not  in  business)  are  all 
unavailable  under  the  current  statutory  and  rule  structure. 

I  do  not  want  anyone  to  misunderstand  anything  written  in  this  article;  I  am  a 
strong  believer  in  formal  education,  including  education  in  business.  My  second  and 
third  master’s  degrees  are  in  contract  and  acquisition  management  from  the  Florida 
Institute  of  Technology  and  national  security  management  from  the  National  Defense 
University’s  Industrial  College  of  the  Armed  Forces,  respectively.  However,  I  believe 
even  more  strongly  that  we  should  hire  the  best  and  brightest,  no  matter  their  degree 
areas,  and  then  train  them  to  the  specific  knowledge,  skills,  and  abilities  that  they 
need  to  do  federal  government  contracting,  which  is  not  directly  what  they  would 
have  learned  in  most  business  degree  programs. 


CURRENT  SITUATION:  THE  GRAYING  WORKFORCE 

There  is  little  doubt  that  we  have  a  graying  acquisition  workforce.  I  note  that  every 
morning  when  I  look  in  the  mirror.  In  point  of  fact,  the  entire  federal  workforce  is 
aging.  Although  this  paper  will  specifically  address  GS-1 102  personnel,  particularly 
those  in  the  DoD,  many  other  occupational  series  have  larger  percentages  of 
retirement  eligibles  (e.g.,  GS-340  program  management,  GS-343  management  and 
program  analysis,  GS-51 1  auditing).  Figure  1,  from  GAO-01-509  (2001),  provides 
more  detailed  information,  albeit  somewhat  dated. 
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OCCUPATIONS  PERCENTAGE  ELIGIBLE  TO  RETIRE 


GS-855  Electronics  Engineering 

GS-861  Aerospace  Engineering 

GS-850  Electrical  Engineering 

GS-1801  General  Inspection,  Investigation, 
and  Compliance 

GS-334  Computer  Specialist 

GS-1102  Contracting 

GS-511  Auditing 

GS-1301  General  Physical  Science 

GS-602  Medical  Officer 

GS-110  Economist 

GS-512  Revenue  Agent 

GS-1 101  General  Business  and  Industry 

GS-343  Management  and  Program  Analysis  38 


GS-391  Telecommunications 

GS-301  Miscellaneous  Administration  and 
Program  Series 

GS-201  Personnel  Management 

GS-90  Security  Administration 

GS-856  Electronics  Technician 

GS-  801  General  Engineering 

GS-1 811  Criminal  Investigating  Series 

GS-802  Engineering  Technician 

GS-341  Administrative  Officer 

GS-1 320  Chemistry 

GS-1 822  Mine  Safety  and  Health 

GS-1 05  Social  Insurance  Administration 

GS-360  Equal  Employment  Opportunity 
Compliance 

GS-1 341  Meteorological  Technician 

GS-1 310  Physics 

GS-340  Program  Management 

0  10  20  30  40  50  60 

Source:  GAO  analysis  of  CPDF  data 

FIGURE  1.  ESTIMATED  PERCENTAGE  OF  SELECTED  MISSION-CRITICAL 
OCCUPATION'S  FISCAL  YEAR  1  998  WORKFORCE  THAT  WILL  BE  ELIGIBLE 
TO  RETIRE  AS  OF  THE  END  OF  THE  FISCAL  YEAR  2006 
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In  a  recent  article  in  the  Baltimore  Sun  (2006),  Melissa  Harris  updates  this 
information.  During  his  confirmation  hearings  to  become  the  Administrator  of  the 
Office  of  Federal  Procurement  Policy,  David  Safavian  also  expressed  his  concerns 
about  the  aging  acquisition  workforce. 

The  wave  of  federal  workers  originally  hired  to  spy  on  the  Soviet 
Union,  launch  the  Great  Society,  and  regulate  everyone  from 
polluters  to  drug  makers  in  the  1960s  and  1970s  is  beginning  to  age 
out  of  the  work  force;  an  exodus  that  some  officials  say  could  drain 
expertise  and  diminish  the  quality  of  service. 

The  numbers  point  to  what  some  call  a  retirement  tsunami-.  60 
percent  of  federal  workers  are  older  than  45,  and  many  could  retire 
now  if  they  wanted  to,  compared  with  3 1  percent  in  the  private 
sector,  according  to  one  think  tank. 

Experts  say  that  the  next  five  years  could  see  a  mass  exit  of 
experienced — and  loyal — employees  at  a  time  when  some  younger 
workers  see  public  service  as  a  steppingstone  to  lucrative  private- 
sector  jobs  (Harris,  2006). 

During  his  Senate  confirmation  hearing  less  than  a  year  ago,  federal  procurement 
chief  David  Safavian  said  shaping  a  workforce  of  top-notch  acquisition  professionals 
was  one  of  his  top  priorities.  Recruitment  is  especially  critical  because  roughly  40 
percent  of  senior  contracting  officers  are  eligible  to  retire  in  less  than  five  years.  “We 
seem  to  be  losing  more  folks  than  we  are  bringing  in  right  now,  and  that’s  a  grave 
concern,”  he  said.  (Gruber,  2005). 

The  Federal  Acquisition  Institute  (FAI)  provides  us  with  data  specific  to  the  GS- 
1102  occupational  series:  “Retirement  eligibility  in  the  Contracting  Series  (GS- 
1102)  rose  from  10  percent  in  FY  2002  to  12  percent  in  2004.  However,  retirement 
eligibility  for  full  retirement  climbs  to  30  percent  in  2009  and  5 1  percent  in  2014” 
(2005,  p.  vii). 

The  potential  problem  may  be  much  worse  than  just  trying  to  backfill  for  retirees. 
In  commenting  on  this  kind  of  information,  particularly  that  presented  by  Mr. 
Safavian,  Professor  Steven  L.  Schooner  (2005),  Co-Director  of  the  Government 
Procurement  Law  Program  at  the  George  Washington  University  Law  School, 
commented: 

Yet  even  that  stark  language  undersold  the  extent  of  the  problem. 

Safavian  failed  to  acknowledge  that  (a)  the  acquisition  workforce 
was  insufficient  before  Sept.  11,  2001,  and  (b)  although  Government 
procurement  spending  has  increased  dramatically  since  then,  neither 
the  Office  of  Federal  Procurement  Policy  (OFPP)  nor  Congress  has 
expressed  any  interest  in  commencing  a  meaningful  dialogue  on 
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TABLE  1. 

BACHELOR'S  DEGREES  EARNED  BY  FIELD:  1980  TO  2003 
(The  new  Classification  of  Instructional  Programs  was  introduced  in 
2002-2003.  Data  for  previous  years  has  been  reclassified  where 
necessary  to  conform  to  the  new  classifications.  Based  on  survey.) 


FIELD  OF  STUDY 

1980 

1990 

2000 

2001 

2002 

2003 

Total 

929,427 

1,051,344 

1,237,875 

1,244,171 

1,291,900 

1,348,503 

Agriculture  and  Natural  Resources 

22,802 

12,900 

24,238 

23,370 

23,331 

23,294 

Architecture  and  Related  Services 

9,132 

9,364 

8,462 

8,480 

8,808 

9,054 

Area,  Ethnic,  Cultural,  and  Gender  Studies 

2,840 

4,447 

6,212 

6,160 

6,390 

6,629 

Biological  and  Biomedical  Sciences 

46,190 

37,204 

63,005 

59,865 

59,415 

60,072 

Business 

186,264 

248,568 

256,070 

262,515 

278,217 

293,545 

Communication,  Journalism,  and  Related  Programs' 

28,616 

51,572 

57,058 

59,191 

64,036 

69,792 

Computer  and  Information  Sciences 

11,154 

27,347 

37,788 

44,142 

50,265 

57,439 

Education 

118,038 

105,112 

108,034 

105,458 

106,295 

105,790 

Engineering  and  Engineering  Technologies 

69,387 

82,480 

73,419 

72,975 

74,679 

77,267 

English  Language  and  Literature/Letters 

32,187 

46,803 

50,106 

50,569 

52,375 

53,670 

Family  and  Consumer  Sciences/Human  Sciences 

18,411 

13,514 

16,321 

16,421 

16,938 

18,166 

Foreign  Languages,  Literatures,  and  Linguistics 

12,480 

13,133 

15,886 

16,128 

16,258 

16,901 

Health  Professions  and  Related  Clinical  Sciences 

63,  848 

58,  983 

80,863 

75,933 

72,887 

71,223 

Legal  Professions  and  Studies 

683 

1,632 

1,969 

1,991 

2,003 

2,466 

Liberal  Arts  and  Sciences,  General  Studies, 
and  Humanities 

23,196 

27,985 

36,104 

37,962 

39,333 

40,221 

Mathematics  and  Statistics 

11,378 

14,276 

11,418 

11,171 

11,950 

12,493 

Multi/Interdisciplinary  Studies 

11,457 

16,557 

28,561 

27,189 

28,943 

28,757 

Parks,  Recreation,  Leisure  and  Fitness  Studies 

5,753 

4,582 

17,571 

17,948 

18,885 

21,428 

Philosophy  and  Religious  Studies 

7,069 

7,034 

8,535 

8,717 

9,473 

10,344 

Physical  Sciences  and  Science  Technologies 

23,407 

16,056 

18,331 

17,919 

17,799 

17,940 

Psychology 

42,093 

53,952 

74,194 

73,645 

76,775 

78,613 

Public  Administration  and  Social  Services 

16,644 

13,908 

20,185 

19,447 

19,392 

19,878 

Security  and  Protective  Services 

15,015 

15,354 

24,877 

25,211 

25,536 

26,189 

Social  Sciences  and  History 

103,662 

118,083 

127,101 

128,036 

132,874 

143,218 

Theology  and  Religious  Vocations 

6,170 

5,185 

6,789 

6,945 

7,762 

7,926 

Transportation  and  Materials  Moving 

213 

2,387 

3,395 

3,748 

4,020 

4,567 

Visual  and  Performing  Arts 

40,892 

39,934 

58,791 

61,148 

66,773 

71,474 

Other  and  Unclassified 

436 

2,992 

2,592 

887 

388 

147 

1 1ncludes  technologies 

Source:  U.S.  National  Center  for  Education  Statistics,  Digest  of  Educational  Statistics,  annual;  and  unpublished  data. 
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TABLE  2. 

COLLEGE  FRESHMEN— SUMMARY  CHARACTERISTICS:  1970  TO  2004 
(In  percent,  except  as  indicated  [12.8  represents  $12,800].  As  of  fall 
for  first-time  full-time  freshmen  in  4-year  colleges  and  universities. 
Based  on  sample  survey  and  subject  to  sampling  error;  see  source.) 


CHARACTERISTIC 

1970 

1980 

1985 

1990 

1995 

2000 

2002 

2003 

2004 

Probable  field  of  study: 

Arts  and  Humanities 

(n/a) 

10.5 

10.1 

10.5 

11.2 

12.1 

12.6 

12.3 

12.0 

Biological  Sciences 

(n/a) 

4.5 

4.5 

4.9 

8.3 

6.6 

7.2 

7.3 

7.7 

Business 

(n/a) 

21.2 

24.6 

21.1 

15.4 

16.7 

16.2 

15.9 

16.0 

Education 

(n/a) 

8.4 

6.9 

10.3 

10.1 

11.0 

10.6 

10.1 

9.6 

Engineering 

(n/a) 

11.2 

11.0 

9.7 

8.1 

8.7 

9.5 

9.3 

9.6 

Physical  Science 

(n/a) 

3.2 

3.2 

2.8 

3.1 

2.6 

2.7 

2.7 

3.0 

Social  Science 

(n/a) 

8.2 

9.4 

11.0 

9.9 

10.0 

10.4 

10.5 

10.3 

Professional 

(n/a) 

15.5 

13.1 

13.0 

16.5 

11.6 

12.3 

14.3 

15.1 

Technical 

(n/a) 

3.1 

2.4 

1.1 

1.2 

2.1 

1.4 

1.5 

1.5 

Data  Processing/ 

Computer  Programming 

(n/a) 

1.7 

1.7 

0.7 

0.8 

1.5 

0.9 

0.7 

0.6 

Other1 

(n/a) 

14.0 

15.1 

15.8 

16.0 

17.9 

16.9 

16.0 

15.0 

Communications 

(n/a) 

2.4 

2.8 

2.9 

1.8 

2.7 

2.5 

2.4 

1.9 

Computer  Science 

(n/a) 

2.6 

2.4 

1.7 

2.2 

3.7 

2.2 

1.7 

1.4 

(n/a)  -  Not  available 

1 1ncludes  other  fields,  not  shown  separately 

Source:  The  Higher  Education  Research  Institute,  University  of  California,  Los  Angeles,  CA.  The  American  Freshman:  National 
Norms,  annual. 

this  problem.  Thus,  despite  a  clear  need  for  additional  resources, 
Safavian,  like  many  of  his  predecessors  at  OFPP,  steadfastly  refuses 
to  call  for  increasing  the  size  of  the  acquisition  workforce. 


ASSESSING  THE  CANDIDATE  POOL 

Given  this  huge  potential  for  retirements  and  potential  need  for  even  more 
contracting  personnel,  we  must  now  look  at  the  sources  for  those  candidates.  This 
becomes  particularly  important  if  we  assume  that  the  federal  government  will  want 
to  continue  to  maintain  an  organic  capability  for  performing  acquisition  contracting, 
notwithstanding  that  there  will  probably  be  a  need  to  supplement  that  organic 
capability  with  contractor  support.  Because  of  the  inherently  governmental  functions 
associated  with  the  acquisition  contracting  process,  this  should  be  a  safe  assumption. 

The  FAI  (2005)  reports  that  in  both  2003  and  2004,  the  last  years  for  which 
statistics  are  available  currently,  about  twice  the  number  of  contracting  occupational 
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All  Candidates  Who  Meet 
Business  Units  Criterion 


All  Candidates  Who  Meet 
Degree  Criterion 


All  Candidates 


Capable  of  Doing  the 
Job  Based  on  Ability 


All  Candidates  Capable 
of  Doing  the  Job  and  Who 
Meet  Degree  Criterion 


All  Candidates  Capable 
of  Doing  the  Job  and  Who 
Meet  Degree  and  Business 
Units  Criterion 


FIGURE  2.  THE  DIMINISHING  POOL  OF  HIRABLE  CANDIDATES 


series  hires  came  from  internal  hires  and  employees  changing  agencies  than  came 
from  external  hires.  This  “robbing  Peter  to  pay  Paul”  approach  to  recruitment  is  not  a 
long-term  solution,  and  it  may  only  exacerbate  problems  in  the  long  run  by  obscuring 
the  true  significance  of  the  problem.  The  only  viable  long-term  solution  to  developing 
and  maintaining  an  organic  acquisition  contracting  workforce  is  through  external 
hires. 

Let’s  look  at  the  potential  pool  of  future  candidates.  For  DoD,  that  means  college 
graduates  with  a  minimum  of  a  bachelor’s  degree  and  at  least  24  hours  in  business 
or  business-related  courses.  The  preceding  two  tables  from  the  Statistical  Abstract  of 
the  United  States  (2006)  delineate  that  future  pool.  Recognize,  however,  that  the  DoD 
candidate  pool  (i.e.,  college  graduates  with  a  minimum  of  a  bachelor’s  degree  and 
at  least  24  hours  in  business  or  business-related  courses)  is  not  necessarily  reflective 
of  those  potential  candidates  that  are  actually  capable  of  performing  the  job  based 
on  ability.  Additionally,  recognize  that  the  DoD  candidate  pool  contains  people  who 
meet  the  two  criteria,  but  are  not  capable  of  performing  the  job  based  on  ability. 

Although  Table  1  shows  the  absolute  number  of  business  graduates  has  been 
growing,  in  Table  2,  it  is  important  to  note  the  decline  of  the  percentage  of  college 
freshmen  opting  for  business  degrees,  a  decline  of  five  percentage  points  over  the  last 
25  years. 

As  a  result  of  the  narrow  criteria  used,  the  DoD  has  a  limited  pool  of  candidates 
from  which  to  select.  Figure  2  reflects  a  number  of  overlapping  and  intersecting 
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pools,  representing  the  Universe  of  Candidates,  All  Candidates  Capable  of  Doing  the 
Job  Based  on  Ability,  All  Candidates  Who  Meet  Degree  Criterion,  All  Candidates 
Who  Meet  Business  Units  Criterion,  All  Candidates  Capable  of  Doing  the  Job  and 
Who  Meet  Degree  Criterion,  and,  finally,  All  Candidates  Capable  of  Doing  the  Job 
and  Who  Meet  Degree  and  Business  Units  Criteria.  Although  the  size  of  the  circles 
is  notional  and  strictly  arbitrary,  the  important  fact  to  remember  is  that  the  pool  of 
candidates  from  which  DoD  may  select  contracting  personnel  becomes  very  limited 
in  comparison  to  the  total  number  of  candidates  that  may  be  able  to  do  the  job 
effectively  and  efficiently. 


LIMITING  DIVERSITY 

Another  unintended  adverse  consequence  of  limiting  the  pool  by  establishing 
degree  and  course  requirements  is  the  narrowing  of  diversity  in  the  acquisition 
contracting  workforce.  Although  we  may  have  a  tendency  to  think  of  diversity 
in  terms  such  as  age,  disability,  economics,  gender,  ethnicity,  religion,  or  sexual 
orientation,  diversity  also  includes  other  areas,  such  as  education.  In  discussing 
diversity,  R.  Mclnnes  (2006)  writes: 

As  we  enter  the  2 1  st  century,  workforce  diversity  has  become  an 
essential  business  concern.  In  the  so-called  information  age,  the 
greatest  assets  of  most  companies  are  now  on  two  feet  (or  a  set  of 
wheels).  Undeniably,  there  is  a  talent  war  raging.  No  company  can 
afford  to  unnecessarily  restrict  its  ability  to  attract  and  retain  the  very 
best  employees  available. 

Generally  speaking,  the  term  Workforce  Diversity  refers  to  policies 
and  practices  that  seek  to  include  people  within  a  workforce  who  are 
considered  to  be,  in  some  way,  different  from  those  in  the  prevailing 
constituency. 

Mclnnes  (2000)  continues: 

Tumultuous  change  is  the  norm  in  the  business  climate  of  the  21st 
century.  Companies  that  prosper  have  the  capacity  to  effectively 
solve  problems,  rapidly  adapt  to  new  situations,  readily  identify  new 
opportunities  and  quickly  capitalize  on  them.  This  capacity  can  be 
measured  by  the  range  of  talent,  experience,  knowledge,  insight,  and 
imagination  available  in  their  workforces.  In  recruiting  employees, 
successful  companies  recognize  conformity  to  the  status  quo  as 
a  distinct  disadvantage.  In  addition  to  their  job-specific  abilities, 
employees  are  increasingly  valued  for  the  unique  qualities  and 
perspectives  that  they  can  also  bring  to  the  table. 
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FIGURE  3.  AVERAGE  INCOME  INFORMATION  FOR 
A  NORTHERN  VIRGINIA  RESIDENT 


What  does  this  mean  for  DoD?  Mclnnes’s  description  echoes  the  needs  that  we 
have  in  DoD’s  acquisition  contracting  workforce.  Psychologist  Abraham  Maslow 
(1966)  wrote,  “If  the  only  tool  you  have  is  a  hammer,  it  is  tempting  to  treat  everything 
as  if  it  were  a  nail.”  Over  time,  we  are  reducing  and  limiting  the  intellectual  diversity 
of  the  DoD  acquisition  contracting  workforce.  This  is  of  critical  concern  at  a  time 
when  we  are  asking  our  workforce  to  be  more  flexible,  innovative,  and  creative.  In 
years  past,  we  would  have  leveraged  the  diverse  thinking  and  analytical  skills  and 
tools  of  people  with  various  backgrounds  (e.g.,  business,  history,  communication, 
political  science,  chemistry,  and  theology1).  In  the  future,  we  will  bring  the  thinking 
and  analytical  skills  and  tools  of  a  less  diverse  group:  business,  business,  business, 
and  business. 


LAGGING  COMPENSATION 

Yet  another  unintended  adverse  consequence  is  that  it  is  more  difficult  to  compete 
for  qualified  candidates.  In  addition  to  fishing  in  a  smaller  pool  for  candidates, 
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some  of  the  bait  being  used  by  the  federal  government  does  not  compare  well  to  the 
lures  being  used  in  the  private  sector.  We  will  discuss  one,  salary,  although  there  are 
certainly  other  inducements  for  joining  federal  service  (e.g.,  public  service). 

The  ability  of  the  federal  government  to  compete  with  the  private  sector  is 
hampered  by  the  low  initial  salaries  offered.  Unfortunately,  no  additional  money  for 
salaries,  bonuses,  or  pay  differential  came  with  the  higher  entry  requirements  into  the 
contracting  occupational  series.  Figure  3,  from  Salary.com  (2006),  provides  salary 
information  for  a  Buyer  I  in  the  Northern  Virginia  area.  According  to  the  Statistical 
Abstract  of  the  United  States  (2004), 

The  average  beginning  salaries  for  accounting  and  business 
administration/management  graduates  in  2003  were  $36,012  and 
$40,647,  respectively.  By  comparison,  the  2005  Federal  General 
Schedule  salaries  for  entry  level  personnel  at  the  GS-5  and  GS-7 
levels  are  $24,677  and  $30,567,  respectively. 


CRITICALITY  OF  DEGREE  AND  BUSINESS  UNITS 

This  article  has  said  a  good  deal  about  the  unintended  adverse  impacts  of 
requiring  a  bachelor’s  degree  and  24  units  of  business-related  courses.  Flowever,  an 
assessment  is  necessary  to  see  whether  there  may  be  a  critical  importance  to  these 
requirements  that  outweighs  the  unintended  adverse  impacts.  In  his  testimony  before 
the  Acquisition  Advisory  Panel,  Robert  M.  Cooper  (2005)  stated: 

Career  Entry  Standards  should  be  reviewed.  The  loss  of  the  Federal 
Service  Entrance  Examination  (FSEE)  and  subsequent  PACE 
exams  is  a  handicap  in  the  hiring  process.  Intern  Programs  should 
be  examined  for  adequacy,  currency,  and  potential  expansion.  Most 
importantly,  the  24  hours  of  business  course  requirements  should  be 
revisited.  Although  a  background  in  business  education  is  helpful, 
in  itself  it  is  not  a  strong  indicator  of  the  full  range  of  capacities 
required  to  meet  the  challenges  of  risk  management  and  decision 
making  called  for  in  the  expanded  acquisition  role  of  Business 
Manager  imposed  by  recent  Reform. 

Analytical  skill  and  ability  to  clearly  articulate  and  execute  resulting 
weighed  risks  must  be  developed,  identified,  and  verified  before  the 
complex  and  demanding  responsibilities  of  acquisition  personnel 
against  the  scope  and  pace  of  their  environment  will  be  reliably, 
efficiently,  and  effectively  performed.  Demonstrated  education/ 
training/performance  in  case  study,  logical  analysis,  and  English 
composition  are  equally  as  important  as  finance,  accounting,  and 
logistics  for  professional  level  performance  of  acquisition  duties. 


326 


PROFESSIONALISM  IN  THE  ACQUISITION  CONTRACTING  WORKFORCE 


Interestingly  enough,  his  words  echo  both  the  theme  of  this  article  and  the  results 
of  the  U.S.  Merit  Systems  Protection  Board’s  (MSPB)  1992  report  to  the  President 
and  the  Congress,  Workforce  Quality  and  Federal  Procurement:  An  Assessment.  The 
report  dealt  with  many  issues,  including  trying  to  determine  whether  there  were 
indicators  that  could  be  used  to  determine  success  on  the  job  as  a  contract  specialist. 
During  the  MSPB  study,  senior  procurement  officials  provided  the  researchers  with 
suggestions  for  improving  the  quality  of  the  contracting  workforce,  including  limiting 
hiring  for  entry-level  positions  in  the  GS-1 102  series  to  college  graduates  or  persons 
who  have  completed  at  least  24  hours  of  college  courses  in  related  subjects  (p.  18). 
This  reflects  the  current  situation  in  DoD. 

The  MSPB  researchers  attempted  to  isolate  the  relative  importance  of  a  college 
degree  and  business-related  courses.  In  relation  to  the  business  courses,  the  MSPB 
found  (1992): 

An  analysis  of  the  relationship  between  the  completion  of  these 
courses  and  work  performance  revealed  that  there  is  only  a  minimal 
relationship  between  the  number  of  these  courses  the  person  has 
completed  and  the  various  assessments  of  his  or  her  performance. 

In  a  way  this  finding  is  not  surprising.  According  to  supervisors 
responding  to  our  survey,  the  performance  of  contract  specialists 
often  depends  upon  their  ability  to  think  analytically  and  write 
clearly.  Competence  in  these  abilities  may  be  demonstrated  in  a 
variety  of  courses,  from  science  to  English,  and  not  just  in  those  that 
would  appear  to  be  directly  related  to  procurement  work  (pp.  18-19). 

In  point  of  fact,  the  MSPB  researchers  found  that  the  correlation  to  contracting 
training  courses  indicated  a  stronger  relationship  to  job  success: 

Presumably,  the  more  training  completed  the  better  prepared 
employees  should  be  to  perform  their  jobs  well.  In  fact,  analysis 
using  the  various  ratings  of  performance  that  were  discussed  earlier 
in  this  study  showed  that  there  was  indeed  a  statistically  significant 
relationship  between  the  number  of  training  courses  completed  and 
each  of  the  ratings  of  performance  (pp.  19-20). 

In  relation  to  the  college  degree,  the  MSPB  researchers  did  find  a  correlation. 
However,  they  noted  a  significant  caveat  regarding  whether  a  college  degree 
requirement  should  be  established: 

The  single  best  [quality]  indicator  is  the  education  level  of  the 
workforce.  In  general,  the  more  education  completed  by  a  worker 
the  higher  the  quality  of  his  or  her  work.  This  is  not  to  say,  however, 
that  a  person  must  possess  a  college  degree  in  order  to  be  a  high- 
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TABLE  3. 

AVERAGE  PERFORMANCE  RATINGS  AND  SOURCE  OF  ENTRY  INTO  THE 

GS-1 102  SERIES 


SOURCE 

Supervisory 

Rating 

(Abilities) 

Supervisory 

Rating 

(Tasks) 

Annual 

Performance 

Rating 

Research 

Rating 

Self- 

Rating 

(Abilities) 

Self- 

Rating 

(Tasks) 

Intern  Program 

3.19 

3.76 

4.11 

3.88 

3.37 

3.98 

PACE/FSEE 

3.22 

3.85 

4.14 

3.90 

3.38 

3.99 

Cooperative  Program 

3.16 

3.69 

4.06 

3.88 

3.36 

3.89 

Schedule  B 

3.14 

3.68 

4.10 

3.76 

3.32 

3.86 

Work  Study 

2.96 

3.35 

4.16 

3.8 

3.10 

3.70 

Other  Special  Program 

3.09 

3.60 

4.03 

3.75 

3.26 

3.76 

No  Special  Program 

3.08 

3.65 

4.09 

3.81 

3.27 

3.80 

AVERAGE 

3.12 

3.69 

4.10 

3.82 

3.30 

3.85 

quality  contract  specialist.  The  relationship  between  education  and 
performance  is  not  large  enough  to  indicate  that  possession  of  a 
college  degree  should  be  a  minimum  qualification  for  admission  to 
the  field  (p.  49). 

Consistent  with  Mr.  Cooper’s  testimony  to  the  Acquisition  Advisory  Panel,  the 
MSPB  research  would  appear  to  indicate  that  the  loss  of  the  PACE  and  FSEE  exams 
eliminated  a  good  indicator  of  job  performance.  As  shown  in  Table  3,  PACE/FSEE 
was  the  leading  source  for  career  field  entrants  across  five  of  the  six  indicators  used, 
and  it  was  the  second  highest  in  the  only  category  it  was  not  first  (MSPB,  1992,  p. 
28). 


PRIVATE  SECTOR  APPROACH 

The  DoD  and  the  civilian  agencies  are  constantly  being  told  they  should  run  our 
acquisition  business  more  like  the  private  sector.  So,  what  does  private  industry 
do?  Are  bachelor’s  degrees  and  24  hours  of  business-related  courses  the  threshold 
requirement  for  business? 

A  look  at  Salary.com  is  helpful.  The  hiring  criteria  results  of  three  searches  for 
contractor-type  jobs  in  the  private  sector  were:  Buyer  1 — May  require  a  bachelor’s 
degree;  Buyer  II — May  require  a  bachelor’s  degree;  and,  Buyer  III — May  require 
a  bachelor’s  degree.  Additional  searches  for  positions  in  contracts,  contract 
administration,  subcontracts,  and  price  analysis  sometimes  contained  a  degree 
requirement,  but  in  many  cases  this  was  only  listed  as  a  preference. 

Anecdotally,  this  author  discussed  hiring  criteria  with  a  number  of  sources  for 
background  purposes  off-the-record,  including  a  source  from  one  ofDoD’s  top  10 
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contractors.  Generally,  experience,  including  diversity  of  experience,  appeared  to 
be  the  most  significant  criteria,  with  a  degree  being  a  want,  not  a  must.  All  things 
being  equal,  a  degree  would  be  preferred.  However,  even  when  there  was  a  desire  for 
a  degree,  that  desire  would  be  trumped  by  experience.  Rarely  were  a  degree  and  24 
hours  of  business-related  courses  required. 

Robert  C.  Marshall,  Chairman,  Department  of  Economics,  Pennsylvania  State 
University,  in  his  testimony  before  the  Acquisition  Advisory  Panel  (2005),  echoed  the 
above  when  he  was  asked  a  follow-up  question  by  one  of  the  panelists.  The  following 
interchange  occurred: 

MR.  SCHWARTZ:  And  I  have  the  greatest  respect  for  our  wonderful 
Federal  procurement  work  force,  but  I  feel  that  they’re  overmatched. 

We’re  taking  folks  in  some  cases  with  high  school  degrees  and  some 
on  the  job  training,  many  other  cases  with  college  degrees,  and 
they’re  trying  to  do  what  the  MacKenzie  guys  are  doing,  and  it  seems 
to  me  that  that’s  not  a  formula  for  success.  I  wonder  if  you  have  any 
thoughts  about  that. 

MR.  MARSHALL:  Well,  some  of  the  most  impressive  people  I’ve 
talked  to  in  private  industry,  just  to  add  another  category  to  the  list 
you’ve  mentioned  are  people  who — running  top  level  procurement 
people  in  private  companies  came  up  through  a  community  college 
system  and  have  just,  you  know,  ground  out  their  expertise  through 
running  procurements  and  understanding  procurements  and 
understanding  the  vendor  community  in  that  market.  I  mean,  they 
have  just  made  deep  investments  to  understand  what’s  going  on  to 
deliver  best  surplus  to  their  company. 

The  other  idea  to  keep  in  mind  is  that  the  private  sector  emphasizes  “Hire  for 
Attitude,  Train  for  Skill.”  DoD  should  do  the  same.  This  is  important,  as  the  time 
spent  in  formal  and  on-the-job  training  of  a  member  of  the  acquisition  contracting 
workforce  is  significantly  more  that  in  24  units  of  business  courses. 


CONCLUSION 

The  BLUF  discussed  previously  is  the  conclusion  of  this  article.  Although  it 
may  go  against  conventional  wisdom,  the  data  and  analysis  indicate  that  we  need  to 
reassess  past  decisions  associated  with  professionalizing  the  acquisition  contracting 
workforce.  Those  decisions  were  well  motivated,  but  required  additional  analysis. 
Standards  have  been  established  that  have  not  achieved  what  they  were  intended  to 
do  and  have  created  unintentional  adverse  consequences.  There  are  many  examples 
general  and  flag  officers  and  SESs,  and  of  men  and  women  in  the  ranks  of  leadership 
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and  senior  management,  who  would  not  meet  the  current  criteria  to  believe  that  they 
are  just  exceptions  to  the  rule.  The  rules  must  be  revisited. 

Specifically,  DoD  should  seek  to,  at  a  minimum,  eliminate  the  business-related 
course  requirements,  and  possibly  eliminate  the  bachelor’s  degree  requirement  as 
thresholds  for  entry  into  the  GS-1 102  occupational  series.  The  DoD  approach  should 
become  that  in  addition  to  the  training  currently  provided,  there  is  also  an  emphasis 
on  seeking  a  business  degree  in  addition  to  the  education  that  a  candidate  has  already 
received  as  part  of  the  early  development  of  the  new  contract  specialist  or  contract 
administrator. 

Additionally,  an  examination  conceptually  similar  to  the  PACE  must  be  developed. 
Although  the  PACE  was  eliminated  because  of  biases  that  effected  diversity  in 
the  workforce,  the  MSPB  study  demonstrated  that  it  was  the  best  indicator  of  job 
performance  of  any  of  the  entry  sources  studied. 


Mr.  John  Krieger  is  a  professor  of  contract  management  at 
the  Defense  Acquisition  University  (DAU)  and  director  of  DAU's 
Curriculum  Development  and  Support  Contracting  Center.  He  has  27 
years  of  experience  in  contracting  and  acquisition  in  the  Department 
of  Defense  and  civilian  agencies.  His  most  recent  operational 
assignment  was  as  the  assistant  commander  for  contracts  at  the 
Marine  Corps  Systems  Command.  He  is  a  certified  professional 
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ENDNOTE 

1.  For  those  who  may  be  mentally  scoffing  at  the  inclusion  of  theology,  don’t 

forget  that  the  scientific  method  used  for  complex  problem  solving  came  from  a 
Franciscan  friar. 


333 


DEFENSE  ACQUISITION  REVIEW  JOURNAL 


Image  designed  by  TSgt  James  Smith,  USAF 


334 


CONTRACTING  OUT  PROCUREMENT  FUNCTIONS:  CURRENT  STATUS 


CONTRACTING  OUT 
PROCUREMENT 
FUNCTIONS: 
CURRENT  STATUS 


By  J.  Scott  Williams,  Roland  D.  Kankey,  Billy  R.  Harry, 
and  Alan  S.  Gilbreth 

For  years,  the  Department  of  Defense  (DoD)  acquisition  workforce  has  been 
decreasing,  yet  workload  often  has  not  kept  pace.  This  has  created  a  dilemma 
for  DoD  procurement  organizations  that  many  have  addressed  by  contracting 
out  some  of  the  work.  The  Air  Force  Deputy  Assistant  Secretary  (Contracting),  Mr. 
Charlie  Williams,  sponsored  a  study  to  assess  the  current  status  of  contracting  out 
procurement  functions  within  DoD  and  federal  agencies.  Our  study  determined 
that  government  agencies  display  considerable  variety  in  their  use  of  contractor 
support  for  procurement  functions.  This  article  summarizes  the  current  status 
of  contracting  out  procurement  functions  and  recommends  that  contracting 
managers  retain  a  limited  capability  to  contract  out  to  meet  their  mission 
requirements. 


This  article  examines  the  use  of  contractor  support  to  supplement  government 
contracting  personnel  in  the  completion  of  procurement  functions/activities 
using  an  advisory  and  assistance  services  (AAS)  contract.  When  we  address 
the  issue  of  “contracting  out,”  many  automatically  think  we  are  concerned  with 
contracting  out  the  entire  function.  That  is  not  the  case.  According  to  the  Federal 
Acquisition  Regulation  (FAR),  many  procurement  functions/activities  are  inherently 
governmental  (IG).  In  other  words  IG  functions  must  be  performed  by  government 
personnel,  which  include  participating  as  a  voting  member  on  any  source  selection 
boards;  approving  any  contractual  documents;  and  awarding,  administering,  and 
terminating  contracts  (FAR  7.503).  The  FAR  succinctly  precludes  contracting  out 
these  activities:  “Contracts  shall  not  be  used  for  the  performance  of  inherently 
governmental  functions”  (FAR  7.503). 
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POLICY  REVIEW 

It  has  been  a  long-term  policy  of  the  Executive  Branch  to  rely  on  contractors  in 
the  private  sector  to  provide  the  goods  and  services  needed  to  act  on  the  public’s 
behalf  (General  Accounting  Office  [GAO],  1981).  Previously,  the  Office  of  Federal 
Procurement  Policy  (OFPP)  Policy  Letter  92-1  stated,  “inherently  governmental 
functions  necessarily  involve  the  exercise  of  substantial  discretion,”  which  “must  have 
the  effect  of  committing  the  Federal  Government  to  a  course  of  action  when  two  or 
more  alternative  courses  of  action  exist.”  Alternately,  the  FAR  2.101  describes  an  IG 
function  as  follows: 

Inherently  governmental  function  means,  as  a  matter  of  policy, 
a  function  that  is  so  intimately  related  to  the  public  interest  as  to 
mandate  performance  by  Government  employees.  This  definition  is  a 
policy  determination,  not  a  legal  determination. ...  (FAR,  2005) 

Guidance  on  IG  functions  was  reiterated  in  the  Office  of  Management  and  Budget 
(OMB)  Circular  A-76  when  it  was  revised  in  2003,  officially  superseding  OFPP  92-1. 
The  circular  tightened  the  description  of  substantial  discretion.  It  stressed  that  not 
every  exercise  of  discretion  is  substantial.  To  quote  OMB: 

The  use  of  discretion  shall  be  deemed  inherently  governmental 
[substantial  discretion]  if  it  commits  the  government  to  a  course 
of  action  when  two  or  more  alternative  courses  of  action  exist  and 
decision  making  is  not  already  limited  or  guided  by  existing  policies, 
procedures,  directions,  orders,  and  other  guidance  that  (1)  identify 
specified  ranges  of  acceptable  decisions  or  conduct  and  (2)  subject 
the  discretionary  authority  to  final  approval  or  regular  oversight  by 
agency  officials.  (OMB  C  A-76,  2003) 

If  a  function  is  determined  to  be  IG  (e.g.,  the  procurement  function),  but  some 
parts  of  the  work  (activities)  are  noninherently  governmental,  then  these  activities 
could  be  contracted  out  as  AAS  in  accordance  with  FAR  37.2  and  Defense  FAR 
Supplement  (DFARS)  237.2.  The  definition  of  AAS  is  found  in  FAR  2.101(b),  which 
states  in  part: 

Advisory  and  assistance  services  means  those  services  provided 
under  contract  by  nongovernmental  sources  to  support  or  improve: 
organizational  policy  development;  decision-making;  management 
and  administration;  program  and/or  project  management  and 
administration;  or  research  and  development  activities.  (FAR,  2005) 
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A  great  deal  of  information  is  available  on  IG  and  the  contracting-out  decision. 

See  the  referenced  Defense  Acquisition  University  (DAU)  Research  Report  for  more 
historical  detail  and  discussion.  With  respect  to  current  policy,  the  FAR  defines  IG 
in  FAR  2.1,  lists  functions  normally  and  not  normally  considered  IG  and  normally 
considered  IG  in  FAR  7.5,  and  prescribes  policies  and  procedures  to  ensure  that 
IG  functions  are  not  performed  by  contractors  in  FAR  7.5.  The  DoD  has  chosen 
to  supplement  FAR  7.503  with  some  additional  guidance  in  DFARS  207.503  with 
respect  to  those  functions  close  to  IG. 

Another  factor  in  the  decision  as  to  whether  government  personnel  or  contractors 
should  perform  specific  activities  deals  with  breadth  of  vision.  The  GAO  (1991) 
indicated  that  government  decision-making  power  means  more  than  just  being 
the  final  authority  or  signing  the  document.  Government  officials  should  be  active 
throughout  the  decision-making  process.  The  GAO  related  that  the  question  often 
presented  to  courts  was  not  whether  the  contractor  can  be  involved,  but  to  what 
extent  can  the  contractor  be  involved.  Per  the  GAO,  a  key  criterion  was  whether 
the  government  maintains  sufficient  in-house  capability  to  be  thoroughly  in  control 
of  the  policy  and  management  functions.  It  can  be  noted  that  OMB  (2003)  now 
calls  for  agencies  to  consider  the  ability  of  senior  management  to  develop  and 
consider  options  before  contracting  out  activities.  If  contracting  out  is  judged  to 
inappropriately  restrict  this  ability,  then  one  may  be  transferring  IG  authority  to  a 
contractor. 

To  conclude  this  section,  it  should  be  noted  that  the  FAR  echoes  OFPP  Policy 
Letter  92-1  and  specifically  calls  out  several  procurement  activities  as  IG.  This 
clearly  establishes  that  the  entire  contracting  function  cannot  be  contracted  out.  On 
the  other  hand,  the  government  policy  of  dependence  on  the  commercial  sector  to 
the  maximum  extent  for  services  has  been  in  place  for  decades.  Activities  within 
procurement  that  are  considered  to  be  noninherently  governmental  are  legitimate 
candidates  for  contracting  out  under  AAS,  consistent  with  the  guidance  in  FAR 
37.203  and  DFARS  207.503. 

RESEARCH  METHODOLOGY 

The  research  involved  personal  and  telephone  interviews  with  numerous 
contracting  personnel  from  the  military  services,  federal,  and  DoD  agencies.  These 
interviews,  along  with  an  extensive  literature  review,  allowed  development  of  a 
survey  using  a  Microsoft  Word  form.  The  survey  specifically  applied  to  contracting- 
out  duties  performed  by  the  Office  of  Personnel  Management  (OPM)  1102  job 
series/and  equivalent  military  occupational  codes  and  was  not  associated  with  support 
roles — i.e.,  administrative  or  statistical  support.  The  sponsor  (Air  Force  Deputy 
Assistant  Secretary  [Contracting])  helped  establish  links  to  knowledgeable  focal 
points  throughout  the  military  services  who  could  properly  respond  to  the  survey  for 
their  organization.  Building  on  this  start,  the  research  team  used  personal  contacts, 
the  Internet,  published  directories,  and  other  sources  to  contact  other  DoD  and 
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federal  agencies,  soliciting  them  to  likewise  identify  knowledgeable  focal  points  and 
ultimately  knowledgeable  personnel  who  could  provide  meaningful  responses  with 
respect  to  contracting  out  procurement  functions  in  their  organization.  This  approach 
can  be  called  “purposeful”  or  “snowball”  sampling  (McMillan,  1989;  Trochim, 
2001).  While  this  approach  results  in  a  smaller  sample  than  random  sampling  and 
reduces  the  statistical  analysis  of  results,  it  was  the  only  practical  and  efficient 
sampling  approach  for  the  research  question,  because  we  needed  knowledgeable 
experts  from  each  organization,  but  we  did  not  want  multiple  responses  from  an 
organization. 

Once  the  focal  point  for  an  organization  had  been  identified,  a  two-stage 
process  was  utilized.  The  first  contact  tried  to  ensure  that  the  representative  was 
knowledgeable  and  willing  to  participate  in  the  research.  It  is  very  important  to  note 
that  those  who  received  the  survey  had  already  been  identified  as  knowledgeable 
about  contracting  activity  in  their  organization.  The  intent  was  for  each  survey  to 
address  the  status  in  a  separate  organization.  In  some  cases,  one  representative 
possessed  the  requisite  knowledge  and  experience  to  answer  for  a  large  organization 
such  as  the  Department  of  State.  In  other  cases,  such  as  the  National  Aeronautics 
and  Space  Administration  and  the  Department  of  the  Navy,  the  organizational 
representatives  were  located  at  the  regions  and  major  commands  respectively. 
Following  a  positive  first  contact,  the  survey  was  forwarded  as  an  e-mail  attachment. 
While  the  purpose  was  to  get  equal  representation  and  responses  from  each  military 
service,  the  interest  from  the  Air  Force  and  Navy  is  reflected  in  the  responses 
received  when  compared  to  the  lower  Army  response.  Emphasis  was  not  placed  on 
any  one  particular  service  or  agency,  because  the  intent  was  to  represent  the  extent 
of  participation  in  contracting  out  of  procurement  functions.  The  number  of  surveys 
received  reflected  the  responsiveness  of  the  services  and  agencies  contacted.  Once 
completed,  the  form  was  returned  to  the  research  team,  where  it  was  sorted  into 
an  appropriate  grouping  for  the  organization  (service,  DoD  agency,  etc.).  Discrete 
responses  were  tabulated,  and  comments  were  grouped  by  question  and  analyzed. 


RESULTS 

A.  RESPONDENTS 

The  survey  phase  started  in  December  2004  and  was  terminated  on  May  5,  2005. 
As  of  that  date,  57  completed  surveys  had  been  received  from  organizations  within 
the  agencies  in  Table  1 . 

B.  ORGANIZATIONS  CONTRACTING  OUT  PROCUREMENT  SERVICES 

Of  the  57  respondents,  26  indicated  that  their  organizations  were  contracting  out 
for  procurement  services,  25  said  their  organizations  were  not  contracting  out  for 
procurement  services,  5  said  not  currently — but  they  had  plans  to  do  so  in  the  future, 
while  one  indicated  not  currently — but  they  had  in  the  past.  The  distribution  of  those 
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TABLE  1. 

DISTRIBUTION  OF  RESPONDENTS  AMONG  MAJOR  GROUPINGS 


AGENCY 


RESPONSES 


Air  Force 

23 

40 

Army 

4 

7 

Navy 

16 

28 

Other  DoD  Agencies 

8 

14 

Non-DoD  Federal  Agencies 

6 

11 

TABLE  2. 

DISTRIBUTION  OF  RESPONDENTS  WHOSE  ORGANIZATIONS  WERE 
CONTRACTING  OUT  PROCUREMENT  FUNCTIONS 


AGENCY 


RESPONSES 


Air  Force 

12  of  23 

52 

Army 

1  of  4 

25 

Navy 

1  of  16 

6 

Other  DoD  Agencies 

8  of  8 

100 

Non-DoD  Federal  Agencies 

4  of  6 

67 

respondents  who  said  they  were  contracting  out  procurement  services  is  shown  in 
Table  2. 

Based  on  our  process  and  our  sampling  procedure,  it  would  be  unfair  to  impute 
these  percentages  across  all  these  agencies,  but  it  is  interesting  to  note  that  the 
dominant  Navy  response  was,  “we  have  not  done,  nor  do  we  plan  to  do  any 
contracting  out  of  procurement  functions,”  and  the  Army  responses  send  a  similar 
message.  It  is  also  interesting  that  other  DoD  agencies,  other  federal  respondents,  and 
the  Air  Force  indicated  they  were  contracting  out  at  a  higher  rate  than  the  Army  and 
Navy. 

C.  REASONS  FOR  NOT  CONTRACTING  OUT 

For  those  who  indicated  their  organization  was  not  and  had  no  plans  to  contract 
out  any  procurement  functions,  it  was  important  to  know  why.  Some  options  that  had 
been  identified  through  the  literature  search  and  interviews  were  provided.  There  were 
26  respondents  for  this  question.  The  results  are  in  Table  3. 

It  is  interesting  to  note  that  while  the  highest  response  was  “contracting  is  an 
inherently  governmental  function,”  running  a  close  second  was  “current  manpower 
resources  are  sufficient  and  qualified.”  We  solicited  comments  from  our  respondents 
on  each  question.  These  comments  were  to  elaborate  on  their  selected  response(s) 
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TABLE  3. 

WHAT  ARE  THE  MAJOR  REASONS  YOU  ARE  NOT  CONTRACTING  OUT 
PROCUREMENT  SERVICES?  (N  =  26)  (MULTIPLE  RESPONSES  ALLOWED) 


Contracting  is  an  inherently  governmental  function 

15 

Current  manpower  resources  are  sufficient  and  qualified 

13 

Concern  with  handling  proprietary  information 

8 

Could  negatively  impact  competition 

7 

It  is  not  cost  efficient 

4 

Unsatisfactory  contractor  performance 

2 

Other 

6 

and/or  to  raise  other  concerns.  Several  respondents  on  this  question  stated  that  they 
would  contract  out  some  procurement  functions  if  government  resources  became 
inadequate.  The  reservation  some  units  have  with  contracting  out  procurement 
services  seems  to  weaken  when  faced  with  the  task  of  performing  procurement 
services  in  understaffed  conditions.  Other  concerns  identified  by  our  respondents 
were  varied.  One  indicated  that  any  organization  that  contracts  out  loses  control 
of  itself  and  its  future.  The  wisdom  of  contracting  out  was  questioned,  along  with 
the  legality.  Another  specified  that  when  contracting  by  negotiation  (FAR  1 5),  the 
procurement  function  becomes  inherently  governmental  due  to  the  managerial  and 
business  decisions  that  must  be  made.  Others  indicated  that  purely  administrative 
functions  could  be  contracted  out,  and  that  contracting  out  might  be  acceptable 
in  a  surge  situation.  Another  related  that  contracting  out  procurement  functions 
puts  additional  responsibilities  on  the  contracting  officer.  Along  with  the  usual 
responsibilities,  the  contracting  officer  would  need  to  ensure  that  decisions  supported 
by  the  work  of  a  contractor  were  free  from  conflicts  of  interest.  One  was  concerned 
with  the  need  to  maintain  a  pipeline  of  well-trained  and  qualified  1 1 02s  to  assure  a 
viable  cadre  for  movement  to  contracting  officer  positions  in  the  future. 

D.  PROCUREMENT  SERVICES  CONTRACTED  OUT  (PRESENT  AND 
PROJECTED) 

The  next  question  on  the  survey  (Table  4)  attempted  to  ascertain  which 
procurement  functions  were  most  commonly  being  contracted  out.  Thirty-one 
individuals  indicated  their  organization  either  was  contracting  out  procurement 
functions  or  planned  to  do  so  in  the  future.  The  nine  responses  coded  “other” 
involved  three  dealing  with  the  administration  of  construction  contracts,  one  involved 
procurement  training,  and  the  others  were  clarifications  or  qualifications  of  the  listed 
responses.  It  is  important  to  see  where  contractor  support  occurs  in  the  general  flow 
of  the  procurement  process. 

The  data  show  that  contractors  perform  duties  across  the  spectrum  of  procurement 
functions,  both  pre-award  and  post-award.  It  should  be  noted  that  contract  closeout  is 
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TABLE  4. 

WHAT  PROCUREMENT  SERVICES  ARE  YOU  OR  WILL  YOU  BE 
CONTRACTING  OUT?  RANKED  (N  =  31)  (MULTIPLE  RESPONSES  ALLOWED) 


Preparing  contracts  for  closeout 

24 

Performing  price  and  cost  analysis 

18 

Providing  assistance  in  developing  a  statement  of  work 

17 

Market  research 

15 

Drafting/developing  price  negotiation  memorandum 

15 

Receiving/assessing  offers  and  preparing  packages  for  negotiation 

14 

Procurement  planning 

14 

Recommending  a  procurement  strategy  (contract  type) 

14 

Drafting  solicitation  document 

14 

Issuing  solicitation  package 

12 

Processing  award  decision  and  distributing  contract 

11 

Reviewing  performance  and  advising  the  exercise  of  options 

10 

Investigating  reports  of  discrepancy 

10 

Identifying  orders  for  expedited  delivery 

8 

Negotiating  contract  modifications 

8 

Negotiating  price,  terms,  and  conditions 

4 

Other 

9 

the  function  most  heavily  performed  by  contractors.  Traditionally,  this  is  a  function 
that  offices  seek  to  contract  out  due  to  backlogs  and  attention  placed  on  other  higher- 
priority  procurement  functions.  In  contrast,  very  few  are  using  contractor  support 
in  negotiating  price,  terms,  and  conditions.  Negotiation  is  a  function  viewed  by 
most  as  IG  in  nature.  Respondents’  comments  lead  to  the  conclusion  that  in  those 
organizations  where  contractors  perform  some  tasks  related  to  the  negotiation 
function,  government  contracting  officers  perform  the  IG  tasks.  The  responses 
and  comments  reflect  that  contractors  are  tasked  to  perform  functions  across  the 
procurement  spectrum. 

E.  REASONS  FOR  CONTRACTING  OUT 

The  spectrum  of  possible  answers  was  developed  based  on  interviews  with 
Headquarters,  Air  Force  Materiel  Command;  Headquarters,  Defense  Logistics 
Agency;  Defense  Supply  Center-Columbus;  Defense  Supply  Center-Richmond;  and 
the  literature  review.  Results  are  shown  in  Table  5. 

The  dominant  reasons  for  contracting  out  are  centered  on  the  organization’s 
workload.  Two  situations  apparently  drive  the  decision  to  contract  out  portions  of 


341 


DEFENSE  ACQUISITION  REVIEW  JOURNAL 


TABLE  5. 

WHY  ARE  PROCUREMENT  SERVICES  BEING  CONTRACTED  OUT? 
(N  =  31)  (MULTIPLE  RESPONSES  ALLOWED) 


To  meet  workload  surge  requirements 

19 

Inability  to  hire  adequate  resources  to  meet  workload 

18 

Contracting  out  is  faster  than  hiring  to  meet  workload 

11 

Ability  to  select  specific  expertise  required 

11 

Bridge  to  hiring  permanent  employees 

7 

More  cost  effective 

4 

Other 

6 

the  procurement  function — a  temporary  workload  surge  or  a  permanent  increase 
in  workload  where  contractor  employees  are  needed  to  fill  the  gap  until  permanent 
government  employees  can  be  hired.  The  respondents  indicate  that  contracting  out 
is  both  faster  than  hiring  government  employees  and  that  contractors  offer  the  added 
ability  of  being  able  to  provide  the  specific  expertise  required.  The  literature  and 
interviews  with  government  managers  point  out  the  increased  flexibility  contractors 
provide  versus  government  personnel  systems.  The  speed  in  hiring  enjoyed  by  the 
contractor  and  the  ability  of  the  contractor  to  provide  specific  expertise  were  noted  by 
government  procurement  managers. 

F.  PERCENT  OF  PROCUREMENT  WORKFORCE  PROVIDED  BY  THE 
CONTRACTOR 

The  proportion  of  the  workforce  provided  by  the  contractor  was  also  of  interest 
for  this  research.  The  percentages  from  the  respondent  organizations  are  shown  in 
Table  6. 


TABLE  6. 

WHAT  PERCENT  OF  YOUR  CURRENT  PROCUREMENT  WORKFORCE  (1 1 02 
OR  EQUIVALENT)  CONSISTS  OF  CONTRACTOR  EMPLOYEES?  (N  =  31) 


Less  than  1  % 

9 

At  least  1  %,  but  less  than  5% 

3 

At  least  5%,  but  less  than  1 0% 

4 

At  least  1 0%,  but  less  than  20% 

5 

At  least  20%,  but  less  than  40% 

4 

At  least  40%,  but  less  than  60% 

2 

No  response 

4 
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The  mode  was  1%  or  less,  while  the  median  response  was  at  least  5%  but  less  than 
10%.  The  results  show  that  more  than  50%  of  the  respondents  had  less  than  10% 
contractor  employees  in  their  contracting  workforce.  Six  respondents  reported  20  to 
60%  contractor  employees  in  their  workforce.  Those  organizations  having  contractor 
employees  as  a  large  percent  of  their  total  workforce  were  contracting  organizations 
in  Iraq  and  several  smaller  DoD  agencies  in  the  Washington,  DC,  area. 

G.  PROCUREMENT  FUNCTIONS  CONSIDERED  INHERENTLY 
GOVERNMENTAL 

While  the  items  cited  in  the  policy  review  section  provide  some  detail  on  what 
GAO  and  OFPP  felt  were  IG  functions,  the  perspective  of  the  respondents  regarding 
IG  functions  was  also  important.  A  list  of  activities  that  spanned  the  types  of  work 
identified  as  either  IG  or  of  a  type  that  could  cause  concern  if  performed  by  a 
contractor  was  developed  and  respondents  were  allowed  to  select.  Responses  are 
shown  in  Table  7. 


The  3 1  respondents  for  this  question  are  those  who  indicated  their  organization  was 
either  contracting  out  procurement  functions  or  planned  to  do  so  in  the  future.  From 

TABLE  7. 

WHAT  PROCUREMENT  ACTIVITIES  DOES  YOUR  ORGANIZATION 
CONSIDER  INHERENTLY  GOVERNMENTAL? 

(N  =  31)  (MULTIPLE  RESPONSES  ALLOWED) 


Committing  the  government  to  take  some  course  of  action 

30 

Approving  evaluation  criteria 

30 

Terminating  contracts 

30 

Approving  incentive  plans 

29 

Awarding  contracts 

29 

Obligating  funds 

29 

Voting  member  of  the  Source  Selection  Evaluation  Board 

28 

Ordering  changes/taking  action  based  on  contractor  performance 

28 

Determining  if  costs  are  reasonable,  allocable,  or  allowable 

25 

Negotiating  price,  terms,  and  conditions 

24 

Accepting  or  rejecting  services  or  products 

24 

Determining  what  supplies  or  services  are  to  be  acquired 

23 

Use  and  disposition  of  government  property 

23 
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TABLE  8. 

WHERE  DID  YOU  LOOK  FOR  GUIDANCE  REGARDING  POLICY/ 
GUIDELINES  ON  CONTRACTING  OUT  PROCUREMENT?  (N  =  31) 
(MULTIPLE  RESPONSES  ALLOWED) 


Federal  Acquisition  Regulation  and  Supplement 

25 

Organizational  Legal  Office 

24 

Office  of  Federal  Procurement  Policy  (OFPP) 

17 

OMB  Circular  A-76 

16 

Headquarters  Legal  Office 

8 

Other 

8 

their  responses,  it  appears  all  have  a  common  opinion  with  respect  to  committing 
the  government  to  a  course  of  action,  approving  evaluation  criteria,  and  terminating 
contracts.  The  majority  of  respondents  indicated  their  organizations  considered  all  the 
listed  items  to  be  IG.  Clearly,  items  that  require  a  contracting  officer’s  signature  were 
considered  IG.  The  reduced  count  for  some  of  the  items  may  have  been  impacted  by 
the  organization’s  mission.  Some  organizations  may  not  perform  certain  functions, 
which  could  prompt  the  respondent  to  not  select  that  response.  In  addition,  based 
on  overall  survey  results,  analysis  and  staff  activities  were  less  clearly  IG  and  more 
subject  to  a  contracting  out  decision. 

H.  SOURCES  OF  GUIDANCE 

As  one  would  expect,  organizations  relied  most  heavily  on  the  FAR,  their  specific 
agency  FAR  Supplement,  and  their  own  legal  office.  However,  they  also  substantially 
used  the  OFPP  and  the  OMB  A-76  Circular,  which  both  provide  a  good  description 
and  examples  of  what  is/is  not  IG.  Several  organizations  checked  the  “other”  box  and 
spoke  to  evaluating  precedents  in  contracting  out  procurement.  Sources  of  guidance 
for  contracting  out  are  shown  in  Table  8. 

I.  LEGAL  LIMITS/CONCERNS 

Nineteen  of  3 1  respondents  indicated  they  did  not  receive  any  legal  limits  or 
concerns  in  their  guidance.  However,  a  few  specific  concerns  were  brought  to  the 
forefront.  The  first  of  these  was  the  need  for  contractor  personnel  to  stay  clear  of  any 
organizational  conflicts  of  interest.  The  second  was  to  ensure  the  contracts  do  not 
entail  personal  services.  Many  respondents  pointed  out  that  contracting  out  was  only 
done  for  augmentation  purposes.  In  no  way  was  contracting  out  intended  to  displace 
current  federal  civil  service  employees.  Many  organizations’  Federal  Activities 
Inventory  Reform  (FAIR)  Act  submissions  identified  contracting  professionals  as 
criterion  “G,”  which  identifies  IG  positions.  Organizations  felt  that  these  positions, 
though  primarily  governmental  in  nature,  did  include  some  functions  that  were  not 
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IG.  Those  noninherently  governmental  functions  were  those  that  could  be  contracted 
out. 

J.  EXPECTED  FUTURE  INVOLVEMENT 

Although  no  one  can  foresee  the  future,  respondents  with  knowledge  of 
contracting  out  procurement  functions  should  have  an  understanding  that  would 
allow  them  to  reasonably  forecast  future  involvement  by  their  organization.  Their 
projections  are  shown  in  Table  9. 

Of  those  responding  to  the  question,  47  percent  said  they  would  be  increasing  the 
contracting  out  of  their  procurement  services  in  the  future,  20  percent  said  they  would 


TABLE  9. 

HOW  DO  YOU  FORESEE  YOUR  FUTURE  INVOLVEMENT  IN 
CONTRACTING  OUT  PROCUREMENT  SERVICES?  (N  =  32) 


Increasing 

14 

Decreasing 

6 

About  the  same 

10 

No  response 

2 

be  decreasing  contracting  out  of  their  procurement  services  in  the  future,  and  33 
percent  said  they  did  not  expect  their  level  of  contracting  out  to  change. 

Those  organizations  foreseeing  increasing  involvement  in  contracting  out 
procurement  services  attributed  the  anticipated  increase  largely  to  the  result  of  more 
workload  being  placed  on  the  organization  with  limited  resources  available  to  meet 
the  workload.  Conversely,  those  organizations  foreseeing  a  decrease  in  contracting 
out  procurement  services  attributed  those  decreases  to  reductions  in  short-term  surge 
requirements. 


CONCLUSIONS 

Respondents  utilizing  contractor  support  for  traditional  contract  specialist  duties 
most  frequently  reported  a  positive  impact  on  the  mission.  To  a  lesser  degree,  they 
also  cited  increased  flexibility  and  generally  highly  qualified  contractors.  While 
some  reported  negative  experience  with  contracting  out  (Gilbreth,  et  al.,  2005),  it  is 
reasonable  to  conclude  that  AAS  contracts  for  support  of  contracting  organizations 
will  likely  increase  in  the  future.  The  following  specific  conclusions  can  be  drawn: 

■  It  is  reasonable  to  contract  out  noninherently  governmental  functions  or  tasks  when 
an  increased  workload  suddenly  appears,  when  a  requirement  for  extra  workload  is 
only  temporary,  or  when  special  expertise  is  required. 
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■  The  services,  the  DoD,  and  other  federal  agencies  seem  to  be  contracting  out 
similar  procurement  functions,  but  the  DoD  and  other  federal  agencies  report  more 
widespread  use  of  this  alternative. 

■  Most  organizations  use  contractor  support  when  mission  accomplishment  drives 
them  to  make  this  decision. 

■  Most  feel  that  development  of  future  contracting  officers  should  not  be  a  problem  if 
contracted-out  procurement  support  is  at  a  reasonable  level. 

■  Contracting  out  procurement  functions  violates  no  laws  so  long  as  no  IG  functions 
are  contracted  out,  unauthorized  personal  services  contracts  are  avoided,  core 
procurement  capability  is  retained,  and  consistency  is  maintained  with  FAIR  Act 
submissions. 

■  Many  organizations’  ability  to  perform  its  mission  would  be  severely  impacted  if  it 
were  suddenly  unable  to  contract  out. 

■  The  reservation  some  units  have  with  contracting  out  procurement  functions  seems 
to  weaken  when  faced  with  an  understaffed  condition. 

■  The  current  definition  of  IG  and  the  examples  provided  in  the  OMB  Circular  A-76 
and  the  FAR  are  well  constructed  and  provide  appropriate  guidance  while  allowing 
the  application  of  the  business  judgment  that  is  necessary  to  accomplish  the  mission 
in  today’s  changing  environment. 


RECOMMENDATIONS 

While  the  survey  results  reflected  that  contractors  are  primarily  used  to  accomplish 
the  more  administrative  tasks,  a  few  of  the  respondents  used  contractors  to 
accomplish  some  of  the  more  sensitive  procurement  tasks — negotiation  of  price, 
terms,  and  conditions — while  the  contracting  officer  (CO)  made  the  final  decision. 

In  this  type  of  arrangement,  there  must  be  substantial  discussion  between  the 
CO  and  the  contractor  typical  of  the  discussion  that  occurs  between  the  CO  and 
the  government  buyer.  In  essence,  an  atmosphere  bordering  closely  on  personal 
services  could  be  created.  If  one  believes  negotiating  price,  terms,  and  conditions  is 
inappropriate  for  contractors  to  perform,  one  could  issue  guidance  precluding  such, 
but  interpretation  and  enforcement  of  this  type  of  policy  is  always  problematic  in 
its  implementation.  Instead  of  a  restrictive  list  of  do’s  and  don’ts  of  contracting  out 
procurement  functions,  a  better  approach  is  available. 

The  research  team  recommends  that  each  procurement  activity  be  limited  in  the 
percentage  of  its  workforce  that  may  be  contracted  out.  The  appropriate  limitation 
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can  certainly  be  debated  but  the  research  team  recommends  that  in  nonexceptional 
situations,  contractor  employees  should  not  exceed  25  percent  of  an  activity’s  total 
1102  workforce.  This  approach  achieves  several  objectives. 

First,  it  provides  each  activity  the  flexibility  to  use  contractors  to  accomplish  the 
mission  by  quickly  reacting  to  surge  workload  situations  within  its  organization’s 
funding  constraints.  Most  respondents  stated  they  preferred  the  use  of  a  government 
workforce  to  accomplish  the  procurement  function  and  only  used  contractors  when 
necessary  to  meet  the  mission.  While  recognizing  this  preference,  the  respondents 
also  found  it  necessary,  at  times,  to  use  contractors  to  meet  the  mission.  In  fact,  the 
findings  indicate  that  many  of  the  respondents  who  were  not  currently  contracting 
out  any  of  their  procurement  functions  because  “procurement  is  inherently 
governmental”  might  feel  otherwise  if  they  were  confronted  with  a  surge  requirement 
that  exceeded  their  capabilities  and  resulted  in  a  negative  mission  impact.  So 
allowing  the  procurement  activities  some  authority  to  contract  out  when  necessary 
seems  prudent. 


Recommend  that  eath  procurement  activity  be  limited  in  the 
percentage  of  its  workforce  that  may  be  contracted  out 


Secondly,  this  approach  addresses  another  concern  when  contracting  out 
the  procurement  workforce.  By  limiting  contractors  to  25  percent  of  the  total 
procurement  workforce,  a  manager  would  typically  assign  the  contractors  to  the 
lower  priority  and  less  sensitive  tasks.  This  is  logical  because  these  tasks  would  be 
the  ones  that  could  not  be  accomplished  by  the  government  workforce,  thus  providing 
a  need  to  contract  out.  By  limiting  the  total  contractor  workforce  to  25  percent,  the 
assumption  is  that  they  would  be  less  involved  in  the  more  sensitive  procurement 
tasks.  Obviously  there  could  be  exceptions,  but  management  would  make  these 
decisions  only  when  appropriate.  For  instance,  a  contractor  employee  who  has 
extensive  government  contracting  experience  and  is  trusted  by  the  contracting  officer 
(CO)  could  be  used  to  negotiate  price,  terms,  and  conditions. 

Finally,  this  policy  would  also  help  address  the  concern  of  growing  future 
COs.  Some  have  expressed  concern  that  extensive  contracting  out  would  have 
the  long-term  effect  of  reducing  the  opportunity  to  develop  adequate  government 
personnel  who  have  the  full  range  of  contracting  experience  necessary  to  meet 
the  CO  needs  of  the  future.  A  25  percent  limit  on  the  contractor  workforce  should 
provide  management  the  opportunity  to  develop  prospective  COs  in  all  aspects  of 
procurement. 

It  is  also  recommended  that  a  process  be  established  for  situations  when  it  becomes 
necessary  to  exceed  the  suggested  contractor  percentage  limits.  The  process  should 
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not  be  overly  onerous,  but  should  have  an  approval  level  outside  the  procurement 
activity  with  a  specific  time  limit  for  the  waiver.  These  short-term  situations  should 
be  accommodated  and  should  not  have  a  negative  impact  if  well  managed.  The  CO 
function  can  still  remain  governmental,  and  a  short-term  situation  should  not  impact 
the  development  of  future  COs. 
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A  PROPOSAL  FOR  A 
NEW  APPROACH  TO 
PERFORMANCE-BASED 
SERVICES  ACQUISITION 


Vernon  J.  Edwards  and  Ralph  C.  Nash ,  Jr. 


Performance-Based  Services  Acquisition  (PBSA)  is  the  government's  preferred 
approach  for  service  contracting,  but  despite  great  efforts  and  training,  PBSA 
remains  difficult  to  implement.  An  analysis  of  services  acquisition  suggests  that 
while  PBSA  may  be  useful  for  routine,  common,  and  relatively  simple  services, 
it  is  not  as  applicable  for  services  that  are  too  long-term  and  complex  to 
permit  complete  specification  of  results  and  competitive  pricing  at  the  outset 
of  contracting.  A  new  approach  for  contracting  these  kinds  of  services  is 
recommended. 


Since  the  publication  of  the  Office  of  Federal  Procurement  Policy’s  (OFPP)  policy 
letter  91-2,  Service  Contracting,  on  April  15,  1991,  Performance-Based  Services 
Acquisition  (PBSA,  formerly  called  “Performance-Based  Contracting”) 
has  been  the  government’s  preferred  approach  to  service  contracting.  It  requires 
specification  of  the  results  that  contractors  must  produce  instead  of  the  processes  that 
they  must  use. 

Agency  acquisition  managers  and  working-level  agency  acquisition  personnel 
have  devoted  a  lot  of  energy  to  PBSA  since  1991.  But  despite  goal-setting,  the 
publication  of  numerous  guidebooks,  the  development  of  an  informational  Web  site, 
and  significant  investments  in  training  and  in  the  services  of  consultants,  PBSA 
has  not  been  as  successful  as  hoped,  and  agencies  still  struggle  to  apply  it.  Even 
when  agencies  claim  to  have  adopted  PBSA,  close  examination  of  their  contracts 
often  shows  that  those  documents  do  not  entirely  satisfy  the  criteria  in  the  Federal 
Acquisition  Regulation  (FAR)  (GAO,  2002).  Moreover,  despite  occasional  agency 
“success  stories,”  the  policy  has  not  produced  verified  quality  improvements  or  cost 
savings. 1 
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This  article  proposes  that  there  are  two  categories  of  services,  and  that  PBSA  as  it 
is  known  at  the  beginning  of  2007  works  for  one,  but  not  the  other.  The  first  category 
includes  many  common,  routine,  and  relatively  simple  services  that  can  be  acquired 
through  PBSA  as  it  is  currently  defined,  including  many  housekeeping  services, 
simple  equipment  maintenance  and  repair  services,  and  the  like.  The  second  category 
includes  services  that  are  too  long-term  and  complex  to  permit  complete  specification 
of  results  and  competitive  pricing  at  the  outset  of  contracting.  These  include  many 
long-term  information  technology  services,  services  to  operate  government-owned 
facilities,  and  long-term  and  multifunction  or  multitask  professional,  administrative, 
and  management  support  services.  These  are  the  services  for  which  the  government 
spends  the  most  money.  This  article  proposes  a  new  approach  to  contracting  for  this 
second  category  of  services. 


DESCRIPTION  AND  HISTORY  OF  PBSA 

The  FAR  provides  that  when  using  PBSA,  agencies  must  specify  the  service  results 
(outputs,  outcomes)  they  want  in  “clear,  specific,  and  objective  terms  with  measurable 
outcomes.”  They  must  prepare  performance  work  statements  and  quality  assurance 
surveillance  plans,  use  performance  incentives  when  appropriate,  and  inspect  and 
compensate  contractors  on  the  basis  of  their  work  products  rather  than  their  work 
processes. 

The  PBSA,  in  various  manifestations,  has  a  long  history.  During  1969-1971,  the 
Office  of  Economic  Opportunity  (OEO)  in  the  Department  of  Health,  Education, 
and  Welfare  experimented  with  an  outcomes-based  approach  to  contracting  for 
educational  services.  The  results  were  mixed  and  the  program  was  dropped.2 
In  September  1979,  the  U.S.  Air  Force  adopted  a  comprehensive  performance- 
based  approach  to  contracting  for  base  support  services,  which  OFPP  adopted 
for  government- wide  use  in  October  1980.  The  efforts  of  the  Air  Force  and  OFPP 
produced  few,  if  any  positive  results.3 

The  1991  OFPP  policy  letter  was  a  response  to  growing  concerns  about  the 
amounts  that  agencies  were  spending  to  buy  services  and  the  quality  of  the  services 
they  were  receiving.4  However,  agencies  were  slow  to  respond  to  the  policy  letter, 
and  although  the  letter  called  for  FAR  implementation  before  the  end  of  1 99 1 ,  it 
was  not  until  1997  that  the  FAR  was  amended  to  include  rules  for  PBSA.5  Since 
then,  agencies  have  tried  to  use  the  technique,  but  with  disappointing  results. 
Implementation  goals  were  established,  but  not  achieved.  Government  acquisition 
officials  and  industry  representatives  have  expressed  doubts  about  the  success  of 
PBSA,  independent  reviews  have  not  validated  predictions  and  anecdotal  claims 
of  improvements  in  quality  and  reductions  in  cost,  and  people  at  the  working  level 
are  frustrated.  In  200 1  and  2002,  the  Honorable  Angela  Styles,  then- Administrator 
of  OFPP,  told  Congress  that  Performance-Based  Services  Acquisition  had  not  been 
more  successful  because  the  concept  had  not  been  adequately  defined.6  In  July  2003, 
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an  interagency  team  assembled  by  OFPP  recommended  minor  changes  to  the  FAR, 
which  were  published  in  December  2005. 7 

WHY  HAS  PBSA  NOT  BEEN  MORE  SUCCESSFUL? 

The  main  reason  that  PBSA  has  not  been  more  successful  is  that  it  is  not  a 
practical  approach  to  buying  long-term  and  complex  services.  Agencies  have  been 
unable  to  implement  PBSA  as  hoped  because  it  requires  them  to  do  something  that  is 
too  often  impracticable. 

It  is  unrealistic  to  ask  agencies  to  specify  services  at  the  time  of  contract  award 
in  clear,  specific,  objective,  and  measurable  terms  when  future  needs  are  not  fully 
known  or  understood,  requirements  and  priorities  are  expected  to  change  during 
performance,  and  the  circumstances  and  conditions  of  performance  are  not  reliably 
foreseeable.  Yet  those  are  the  difficulties  faced  by  agencies  and  their  contractors  when 
they  negotiate  long-term  and  complex  service  contracts. 


The  main  reason  that  PBSA  has  not  been  more  successful 
is  that  it  is  not  a  practical  approach  to  buying  long-term 
and  complex  services. 


In  real  life,  parties  to  long-term  and  complex  service  contracts  do  not  specify 
all  requirements  at  the  time  of  contract  award  in  clear,  specific,  objective,  and 
measurable  terms.  Instead,  they  engage  in  ad  hoc  decision  making  in  response 
to  emerging  and  changing  requirements,  shifting  priorities,  and  unexpected 
circumstances.  They  make  it  up  as  they  go  along,  developing  and  adjusting 
expectations  and  agreements  accordingly.  Reality  is  never  the  same  as  expectations 
and  projections,  and  plans  and  agreements  go  awry.  No  matter  how  long  and  hard 
future  needs  are  considered,  contracts  will  include  things  that  will  not  be  needed  and 
leave  out  things  that  will  be.  Specifications  and  expectations  must  be  adjusted  over 
the  course  of  time.8 

Thus,  in  requiring  that  agencies  fully  specify  results  at  the  outset  of  contracting, 
PBSA  often  requires  them  to  do  something  that  is  too  hard  to  do  and  sets  them  up 
to  fail.  More  training  will  not  make  PBSA  appropriate  for  long-term  and  complex 
service  acquisitions. 
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THE  CHALLENGES  OF  SERVICE  CONTRACTING 

When  contracting  for  services,  agencies  must  follow  regulations  and  use  practices 
that  were  developed  for  the  procurement  of  supplies.  Supplies  are  always  specified 
based  on  known  design  or  performance  requirements.  The  PBSA  is  an  attempt  to 
buy  services  like  we  buy  supplies.  But  this  attempt  ignores  key  differences  between 
supplies  and  services. 

SERVICE  QUALITY 

Unlike  most  supply  purchases,  long-term  service  contracts  entail  close  human 
relationships  that  enable  the  parties  to  deal  with  dynamic  complexity  and  respond  to 
emerging  and  changing  needs  and  circumstances.  Relationships  are  crucial,  and  it  is 
well  established  in  service  marketing  literature  that  subjective  customer  satisfaction  is 
as  important,  and  sometimes  more  important,  than  technical  success.9  The  importance 
of  subjective  factors  in  government  service  contracting  is  confirmed  by  the  fact 
that  subjective  incentives — award  fee  and  award  term — are  the  most  popular  of  all 
incentives  used  in  performance-based  contracts.10 


Unlike  most  supply  purchases,  long-term  servife  contracts 
entail  dose  human  relationships  that  enable  the  parties  to 
deal  with  dynamic  complexity  and  respond  to  emerging  and 
changing  needs  and  circumstances. 


Services  confront  agencies  with  quality  specification  problems,  unlike  those 
associated  with  contracts  for  supplies.  Services  are  always  rendered  in  response  to 
actual  circumstances  and  conditions.  It  is  often  impossible  and  even  unwise  to  try 
to  fix  specifications  of  service  quality  at  the  outset  of  contract  performance,  because 
quality  often  “depends.”  What  is  good  service  in  one  set  of  circumstances  might  be 
poor  service  in  another,  and  the  standard  contract  modification  process  is  not  nimble 
enough  for  the  realities  and  demands  of  a  high-speed  electronic  world. 

The  PBSA  requirement  for  beforehand  specification  and  objective  and  measurable 
standards  ignores  the  nature  of  long-term  and  complex  service  relationships.11 

CONTRACTOR  SELECTION  AND  CONTRACT  PRICING 

A  lynchpin  of  PBSA  is  competitive  contractor  selection  based  on  price  and  other 
factors  in  compliance  with  the  Competition  in  Contracting  Act  (CICA)  and  FAR  Part 
15.  In  theory,  PBSA  allows  competing  firms  to  devise  their  own  ways  to  produce 
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the  results  sought  by  the  government,  which  supposedly  lets  the  government  enjoy 
the  benefits  of  vigorous  price  competition.  But  when  an  agency  cannot  describe  its 
requirements  and  the  circumstances  and  conditions  of  performance,  competing  firms 
cannot  do  so  either.  So  when  an  agency  evaluates  a  proposal  for  a  service  contract,  it 
evaluates  the  product  of  the  marketing  imagination,  which  describes  something  that 
does  not  yet  exist  and  cannot  be  examined  or  tested  before  purchase. 

An  agency  proposal  evaluation  team  cannot  be  sure  whether  the  firm  selected  for 
contract  award  will  truly  be  the  best  value  or  that  it  just  produced  the  best  proposal 
document.  In  the  absence  of  specific  knowledge  about  future  needs,  firms  cannot 
propose  specific  solutions,  and  strict  enforcement  of  vague  commitments  is  an 
unlikely  prospect.  In  the  absence  of  clear  and  binding  promises,  prices  or  estimated 
costs  are  not  very  meaningful.  Comparative  evaluation  of  competing  proposals  of 
service  quality  and  prices  is  thus  a  dubious  undertaking,  because  an  agency  cannot  be 
sure  about  what  it  will  actually  get  or  be  entitled  to  get  from  a  contractor  for  its  price. 

Contractor  selection  under  FAR  Part  1 5  procedures  does  not  readily  permit  a  full 
and  frank  airing  of  issues  and  resolution  of  concerns  between  the  government  and 
its  contractor  before  contract  award.  Industry  responses  to  draft  solicitations  and 
participation  in  preproposal  conferences  are  constrained  by  competitive  strategy  and 
tactics  and  government  reticence.  After  proposal  submission,  agencies  either  award 
contracts  without  discussions  or  conduct  discussions  that  are  constrained  by  issues 
of  fairness  and  procedure  and  fear  of  protests.  The  result  is  that  the  parties  to  a  new 
contract  are  often  virtual  strangers  to  one  another,  who  learn  of  gaps  and  disconnects 
in  their  understanding  of  the  work  and  their  expectations  only  after  contract  award. 

The  CICA  price  competition  and  FAR  Part  15  source  selection  are  ill-fitted  to  the 
procurement  of  long-term  and  complex  services. 

CONTRACT  ENFORCEMENT:  PRICE  REDUCTIONS,  DAMAGES,  AND 
TERMINATIONS 

The  FAR  tells  contracting  officers  to  inspect  service  results  and  make  price  or 
fee  reductions  when  services  are  not  acceptable.  However,  long-term  and  complex 
services  confront  agencies  with  unique  quality  assurance  problems.  Inspection  can 
be  difficult  because  many  service  results  are  intangible,  and  many  tangible  results  are 
ephemeral.  One  hundred  percent  inspection  is  usually  impracticable,  but  acceptance 
sampling  is  not  always  appropriate.  The  quality  of  some  results,  like  the  results  of 
observational  or  analytical  work,  may  depend  on  the  quality  of  unobservable  mental 
processes.  It  is  easy  enough  to  verify  that  a  floor  is  “clean”  in  the  morning,  that 
wastebaskets  have  been  emptied,  that  grass  has  been  cut  to  a  prescribed  length,  and 
that  an  item  of  equipment  has  been  repaired.  But  the  results  produced  by  security 
guards  who  must  check  the  identities  and  possessions  of  the  hundreds  of  persons 
seeking  entry  to  a  federal  office  building  on  a  daily  basis  are  not  easily  inspected  or 
verified. 

Reviews  of  decisions  by  boards  of  contract  appeals  and  by  courts  about  price 
reductions  under  long-term  and  complex  performance-based  contracts  show  that  price 


357 


DEFENSE  ACQUISITION  REVIEW  JOURNAL 


reductions  generally  are  not  a  satisfactory  remedy  for  poor  performance.  Under  long¬ 
term  and  complex  contracts,  such  reductions  are  administrative  nuisances  to  both  the 
agency  and  to  its  contractor,  and  reductions  for  minor  technical  flaws  in  performance 
sour  a  business  relationship.  Moreover,  price  reductions  and  money  damages  cannot 
adequately  compensate  the  government  for  poor  performance  of  critical  operations. 

Termination  is  truly  a  last  resort  when  a  contract  is  for  long-term  and  complex 
services  because  it  takes  a  lot  of  time  and  effort  to  award  a  replacement  contract, 
and  award  might  be  delayed  by  a  protest.  So  an  agency  might  choose  to  live  with 
marginal  performance,  or  be  forced  to  exercise  an  option  to  extend  a  contractor  that  is 
performing  marginally  so  it  can  buy  time  to  find  a  replacement.  In  sum,  contract  law 
and  court  enforcement  cannot  ensure  satisfactory  service  and  cannot  remedy  poor 
performance.  The  only  way  for  the  government  to  get  the  service  results  it  needs  is 
by  choosing  good  contractors  and  by  establishing  and  maintaining  effective  working 
relationships  with  them.  Relationship  management,  not  contract  administration,  is  the 
key  to  success. 

Contract  law  and  court  enforcement  cannot  guarantee  satisfactory  service  or 
adequate  remedies  for  poor  performance. 


HOW  THE  GOVERNMENT  SHOULD  BUY  LONG-TERM  AND 
COMPLEX  SERVICES:  AN  EMPHASIS  ON  RELATIONSHIPS 

While  the  government  should  usually  focus  on  service  results  instead  of  processes, 
the  realities  of  long-term  and  complex  service  contracting  require  a  new  approach 
to  PBSA.  The  following  paragraphs  describe  a  relational  approach  to  PBSA,  an 
approach  that  emphasizes  the  need  to  establish  a  solid  working  relationship  between 
the  government  and  its  contractor  that  will  allow  the  two  of  them  to  engage  in  ad  hoc 
specification  and  adjustment  of  expectations  throughout  the  life  of  the  contract. 

This  proposed  approach  to  PBSA  is  called  Relational  Contracting  or  Relational 
PBSA.  The  key  features  of  this  approach  are: 

■  competency-based  contractor  selection; 

■  in-depth,  one-on-one  negotiations  with  the  contractor  selectee  before  contract  award 
to  jointly  develop  a  contract  work  statement; 

■  joint  management  to  budget  instead  of  to  a  fixed-price  or  estimated  costs; 

■  advanced  agreement  on  specified  direct  and  indirect  cost  limitations; 

■  ad  hoc  specification  of  results  and  adjustment  of  expectations  during  performance; 

■  fair  and  reasonable  fee  arrangements;  and 


358 


A  PROPOSAL  FOR  A  NEW  APPROACH  TO  PERFORMANCE-BASED  SERVICES  ACQUISITION 


■  mandatory  use  of  alternative  dispute  resolution  procedures. 

An  agency  would  use  the  relational  approach  to  PBS  A  only  when: 

1.  the  contract  will  be  of  at  least  two  years  duration,  including  options; 

2.  the  contract  will  have  a  total  value  of  at  least  $10  million,  including  options; 

3.  the  agency  cannot  fully  specify  key  requirements  or  describe  key  performance 
circumstances  at  the  time  of  contract  award; 

4.  the  head  of  the  contracting  activity  approves  its  use;  and 

5.  the  head  of  the  contracting  activity  makes  provision  for  periodic  independent 
review  of  the  management  of  the  contract  by  neutral  officials. 

This  article  will  now  address  each  of  the  elements  of  relational  PBS  A  in  greater 
detail. 

COMPETENCY-BASED  CONTRACTOR  SELECTION 

The  approach  to  contractor  selection  would  be  similar  to  the  architect-engineer 
selection  approach  described  in  FAR  Subpart  36.6.  Price  would  not  be  a  factor  in 
contractor  selection.  The  main  factors  would  be  experience,  past  performance,  and 
key  personnel  qualifications.  An  evaluation  board  would  consider  candidate  firms  and 
recommend  two  or  three  highly  qualified  firms  to  the  selection  official,  who  would 
then  select  one  for  contract  negotiations. 

The  contracting  officer  would  solicit  an  offer  from  the  selectee,  disclosing  the 
agency’s  budget  and  objectives  and  providing  for  joint  fact-finding  about  known  and 
anticipated  requirements  and  anticipated  performance  circumstances  and  conditions. 
The  parties  would  then  conduct  in-depth  negotiations  to  jointly  develop  a  work 
statement,  an  advance  agreement  on  small  business  subcontracting,  an  advance 
agreement  on  cost  limitations,  and  a  fee  agreement.  The  contracting  officer  would 
award  a  contract  following  approval  of  the  negotiations  in  accordance  with  agency 
procedures. 

This  approach  to  contractor  selection  and  contract  pricing  will  permit  a  fuller 
and  franker  airing  of  issues  and  cooperative  problem  solving  before  contract  award. 

It  will  enable  the  parties  to  reach  a  common  understanding  of  what  they  can  and 
cannot  specify  at  the  outset  and  what  they  must  set  aside  for  ad  hoc  resolution  during 
performance.  This  approach  will  lay  a  better  foundation  for  a  successful  working 
relationship  than  source  selection  under  CICA  and  FAR  Part  15. 
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ADVANCED  COST  LIMITATION  AGREEMENTS/JOINT 
MANAGEMENT  TO  BUDGET 

The  resultant  contract  would  be  a  modified  cost-reimbursement  type,  with 
government  risk  mitigated  by  advance  cost  limitation  agreements.  The  contract 
would  provide  for  the  parties  to  jointly  manage  performance  within  the  government’s 
operation  or  project  budget,  but  with  the  government  having  the  final  say  on  all 
requirements.  The  parties  would  work  together  to  prioritize  and  schedule  activities, 
set  standards,  establish  work  package  budgets,  and  monitor  performance.  They  would 
use  earned  value  management  techniques  when  appropriate. 

AD  HOC  SPECIFICATION  OF  REQUIREMENTS  DURING  PERFORMANCE 

A  key  feature  of  relational  PBSA  would  be  ad  hoc  specification  of  service 
requirements  as  they  emerge  or  become  more  fully  understood  in  the  course  of 
performance.  The  parties  would  specify  requirements  in  terms  of  results  whenever 
possible,  unless  they  agree  that  specification  of  process  would  be  better.  In  order 
to  remain  within  budget,  the  parties  would  make  tradeoffs,  adjusting  expectations, 
reordering  priorities,  and  modifying  performance  standards  as  necessary.  If 
requirements  change,  the  parties  would  bargain  to  make  adjustments  to  stay  within 
the  budget. 


A  key  feature  of  relational  PBSA  would  be  ad  hot 
spedfitation  of  servite  requirements  as  they  emerge  or 
betome  more  fully  understood  in  the  tourse  of  performante. 


Adjustments  within  budget  would  not  require  formal  contract  modifications  and 
equitable  adjustments  and  would  be  within  the  authority  of  the  government’s  program 
manager  as  long  as  they  do  not  require  fund  obligations  or  de-obligations.  But  all 
transactions  would  be  documented  to  reflect  the  agreement  and  expectations  of  the 
parties. 

FAIR  AND  REASONABLE  FEE  ARRANGEMENTS 

The  contract  would  provide  for  payment  of  fee  in  accordance  with  the  agreement 
negotiated  prior  to  contract  award.  The  maximum  available  fee  would  be  fixed  and 
would  not  change  during  the  course  of  performance  unless  the  government  increases 
or  decreases  its  budget  due  to  the  addition  or  deletion  of  requirements.  Changes  in 
budget  due  to  cost  overruns  would  not  entitle  the  contractor  to  additional  fees.  The 
contractor  would  be  guaranteed  a  fair  and  reasonable  fee  for  acceptable  performance 
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within  budget  and  could  earn  additional  fees  for  excellent  performance,  based  on 
objective  and  subjective  considerations  to  which  the  parties  agreed  in  advance. 

MANDATORY  USE  OF  ALTERNATIVE  DISPUTE  RESOLUTION 

The  contract  would  require  the  parties  to  engage  in  alternative  dispute  resolution 
before  resorting  to  the  FAR  disputes  procedures.  Each  party  would  name  one  senior 
official  outside  the  immediate  program  organization  as  its  principal,  and  they  would 
jointly  hear  the  dispute  and  work  to  resolve  it  with  the  assistance  of  a  neutral.  Only 
if  the  two  principals  cannot  agree  on  a  resolution  within  a  reasonable  period  of  time 
would  the  parties  be  permitted  to  resort  to  the  dispute  procedures  described  in  FAR 
Subpart  33.2. 

PREREQUISITES  TO  USE 

Because  relational  PBSA  would  permit  the  award  of  contracts  without  price 
competition,  and  because  it  would  grant  very  broad  discretion  to  government 
program  managers  and  contractor  personnel,  it  is  essential  that  it  be  used  only  when 
appropriate  and  only  as  approved  by  higher  level  agency  officials.  It  is  also  essential 
that  relational  contracts  be  subjected  to  periodic  independent  review  in  order  to 
maintain  the  integrity  of  the  acquisition  system  and  public  confidence.  Relational 
PBSA  should  be  approved  for  use  only  for  complex  contracts  of  two  years  duration 
or  longer  and  with  a  total  value  of  $10  million  or  more,  including  options.  It  is  also 
recommended  that  the  use  of  relational  PBSA  require  approval  of  the  head  of  the 
contracting  activity,  subject  to  arrangements  for  periodic  independent  review  of  each 
relational  contract  by  neutral  agency  officials. 


CONCLUSION 

The  time  has  come  to  try  something  new.  We  propose  that  OFPP  obtain  statutory 
approval  for  a  pilot  program  to  conduct  a  number  of  controlled  experiments  in 
relational  contracting  by  selected  agencies.  The  OFPP  should  set  criteria  for 
evaluating  the  effectiveness  of  relational  contracting,  establish  a  preparatory 
training  program  for  participants,  and  appoint  a  panel  which  includes  executive 
branch  officials;  representatives  of  the  Government  Accountability  Office;  working- 
level  acquisition  personnel;  members  of  academia;  members  of  the  Bar;  industry 
representatives;  and  a  support  staff  to  monitor,  evaluate  and  report  the  results,  and 
make  recommendations  for  further  action. 
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END  NOTES 

1 .  “There  is  little  current  data  to  support  monetary  savings,  and  if  such  data  did 
exist,  it  would  be  extremely  difficult  to  isolate  the  exact  reasons  the  savings 
occurred.”  In  Interagency  Task  Force  On  Performance-Based  Service  Acquisition. 
(2003,  July).  Performance-based  service  acquisition:  Contracting  for  the  future. 
Washington,  DC:  Office  of  Federal  Procurement  Policy,  p.  10.  “The  effect  of 
PBSA  practices  on  contract  prices  is  hard  to  assess  for  the  contracts  we  studied 
because  (a)  the  work  scopes  relevant  to  the  contracts  we  examined  changed  with 
the  new  contracts,  and  (b)  the  Air  Force  has  no  simple  way  to  adjust  costs  for  the 
changes  observed  in  work  scopes.  In  most  cases,  we  could  not  clearly  attribute 
price  changes  to  a  move  toward  PBSA.”  In  Ausink,  J.,  Camm,  F.,  &  Cannon,  C. 
(2001).  Performance-based  contracting  in  the  Air  Force:  A  report  on  experiences 
in  the  field.  Santa  Monica,  CA:  Rand,  p.  34. 

2.  See  Levine,  D.  M.,  ed.  (1971).  Performance  contracting  in  education — An 
appraisal.  Englewood  Cliffs,  NJ:  Educational  Technology  Publications;  and 
Stucker,  J.  P.  &  Hall,  G.  R.  (1971).  The  performance  contracting  concept  in 
education  (R-699/1-HEW).  Santa  Monica,  CA:  Rand.  Performance  contracting 
in  education  was  controversial.  Some  state  and  foreign  governments  still  use  the 
technique  in  secondary  and  higher  education  and,  although  the  results  are  unclear, 
the  technique  still  has  supporters.  Performance  contracting  in  education  should 
not  be  confused  with  energy  performance  contracting. 

3.  The  approach  was  described  in  Air  Force  Regulation  (AFR)  400-28,  Base 
Level  Service  Contracts,  a  detailed,  multi-volume  guide  to  the  preparation 
of  performance  work  statements  and  quality  assurance  surveillance  plans. 
Supplementary  Air  Force  publications,  such  as  AFR  70-9,  Base  Level  Service 
Contract  Administration,  provided  instructions  to  quality  assurance  evaluators 
(inspectors).  OFPP  adopted  the  first  volume  of  that  regulation  for  government¬ 
wide  use  in  October  1980,  dubbing  it  OFPP  Pamphlet  No.  4,  A  Guide  for  Writing 
and  Administering  Performance  Statements  of  Work  for  Service  Contracts.  The 
Air  Force  withdrew  AFR  400-28  in  1994,  replacing  it  and  other  guidance  with 
Air  Force  Manual  (AFMAN)  64-108,  Service  Contracts,  a  63-page  document 
which  still  included  fairly  detailed  guidance  for  the  preparation  of  performance 
work  statements  and  quality  assurance  surveillance  plans.  But  in  1999,  the  Air 
Force  replaced  AFMAN  64-108  with  Air  Force  Instruction  63-124,  Performance- 
Based  Service  Contracts  (PBSC),  an  1 1 -page  document  that  contains  a  statement 
of  policy  but  virtually  no  practical  guidance,  and  it  remains  in  effect  today.  OFPP 
withdrew  Pamphlet  No.  4  in  the  mid-1990s,  but  in  October  1998,  it  issued  A 
Guide  To  Best  Practices  For  Performance-Based  Service  Contracting,  a  severely 
edited  version  of  Pamphlet  No.  4,  containing  less  detailed  guidance.  In  December 
2000,  the  Department  of  Defense  issued  its  current  Guidebook  for  Performance- 
Based  Services  Acquisition  (PBSA)  in  the  Department  of  Defense,  a  54-page 
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document  that  contains  sparse  practical  guidance.  Several  other  agencies  have 
issued  similar  guidance  of  their  own.  Much  of  this  guidance  can  be  accessed 
through  the  Web  site,  Seven  Steps  to  Performance-Based  Services  Acquisition, 
available  at  http://www.amet.gov/Library/OFPP/BestPractices/pbsc/home.html. 
For  a  fascinating  first-hand  account  of  an  attempt  to  develop  a  performance  work 
statement  in  accordance  with  OFPP  Pamphlet  No.  4,  see  Paddock,  C.  D.  (1987, 
June).  Performance  work  statements:  Significant  problems  in  the  preparation 
process.  (Master’s  thesis)  Defense  Technical  Information  Center  (DTIC  No.  AD- 
A184  897). 

4.  “Each  year  the  Government  contracts  for  a  significant  amount  of  services. 

Such  services  range  from  the  routine  maintenance  of  facilities  or  equipment  to 
highly  sophisticated  technical  and  management  assistance  such  as  the  design, 
development  and  furnishing  of  systems,  or  expert  assistance  for  management 
and  program  activities.  Attempts  to  apply  contracting  methods  which  are 
inappropriate  to  the  services  being  acquired  have  often  resulted  in  unsatisfactory 
performance  and  contract  administration  problems,  as  reflected  in  several  internal 
agency  investigations  and  evaluations,  General  Accounting  Office  Reports, 

and  OFPP  studies.  These  reports  criticized  unnecessarily  vague  statements  of 
work,  insufficient  use  of  firmer  pricing  arrangements,  the  lack  of  quantifiable 
performance  standards,  and  the  inadequacy  of  quality  assurance  surveillance.  In 
addition,  there  is  concern  that  the  Government  underemphasizes  quality  versus 
price  in  the  acquisition  of  services.”  In  Office  of  Federal  Procurement  Policy. 
(1991,  April  15).  Policy  letter  on  service  contracting  (56  FR  15110,  15113). 
Washington,  DC:  Author. 

5.  Federal  Acquisition  Circular  97-01,  62  FR  44802,  August  27,  1997. 

6.  “In  part,  I  believe  the  problem  centers  on  a  lack  of  clarity  regarding  the 
definition  of  what  constitutes  a  performance-based  service  contract.  Based  on 
my  experience,  there  is  considerable  disagreement  among  agencies  regarding 
the  requirements  to  qualify  a  contract  as  performance-based.  Previous  attempts 
by  OFPP  to  clarify  the  definition,  including  a  ‘checklist’  of  minimum  required 
elements  for  an  acquisition  to  be  considered  performance-based,  have  been 
unsuccessful.”  Styles,  A.  B.  (2001,  November  1).  Statement  of  Angela  B.  Styles, 
Administrator  for  Federal  Procurement  Policy,  before  the  Subcommittee  on 
Technology  and  Procurement  Policy,  Committee  on  Government  Reform,  United 
States  Blouse  of  Representatives.  Retrieved  from  http://www.acqnet.gov/Notes/ 
sarafinal.doc,  p.  1 1.  See  also  Styles,  A.  B.  (2002,  March  7).  Statement  of  Angela 
B.  Styles,  Administrator  for  Federal  Procurement  Policy,  before  the  Subcommittee 
on  Technology  and  Procurement  Policy,  Committee  on  Government  Reform, 
United  States  House  of  Representatives,  retrieved  from  http://www.acqnet.gov/ 
Notes/saratestimony37.doc 
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7.  Published  results  can  be  seen  in:  Interagency  Task  Force  On  Performance-Based 
Service  Acquisition.  (2003,  July).  Performance-based  service  acquisition: 
Contracting  for  the  future.  Washington,  DC:  Office  of  Federal  Procurement 
Policy;  and  Federal  Acquisition  Circular  (2005-07,  71  FR  198,  21 1),  published 
January  3,  2006. 

8.  This  has  long  been  recognized  in  economic  and  legal  scholarship.  See  Coase, 

R.  H.  (1988).  The  nature  of  the  firm.  In  The  Firm,  the  Market,  and  the  Law. 
Chicago:  The  University  of  Chicago  Press,  pp.  33-56;  Knight,  F.  (1971).  Risk, 
uncertainty  and  profit.  Chicago:  University  of  Chicago  Press;  MacNeil,  I.  R. 
(1978).  Contracts:  Adjustment  of  long-term  economic  relations  under  classical, 
neoclassical,  and  relational  contract  law.  Northwestern  University  Law  Review 
72(6),  p.  854;  and  Williamson,  O.  E.  (1988).  The  economic  institutions  of 
capitalism.  New  York:  The  Free  Press. 

9.  “Of  course,  it  is  possible  to  measure  service  quality  with  more  objective  criteria, 
such  as  in  the  technical  approach  to  quality.  Services  could  be  compared  to  a 
checklist  of  quality  indicators,  such  as  whether  calls  are  answered  in  three  rings 
or  whether  employees  remember  to  smile  and  say  “thank  you”  to  customers 

at  least  99  percent  of  the  time.  However,  setting  specific  goals  for  particular 
aspects  of  service  might  narrow  the  vision  of  employees  so  that  they  will  achieve 
these  goals  by  lowering  quality  in  areas  for  which  no  goals  have  been  set.  For 
example,  service  representatives  might  start  answering  all  customer  calls  within 
three  rings  by  terminating  other  customer  calls  or  placing  people  on  hold.  This 
situation  would  not  be  an  overall  improvement  in  service  quality,  even  though  the 
objective,  technical  approach  to  quality  might  indicate  that  it  was.  Thus  a  user- 
based  approach,  rather  than  an  objective  checklist  approach,  has  been  found  to 
be  superior  for  evaluation  the  quality  of  intangible  services.”  In  Schneider,  B.,  & 
White,  S.S.  (2004).  Service  quality:  Research  perspectives.  Thousand  Oaks,  CA: 
Sage  Publications,  p.l  1.  See,  too,  discussions  of  service  quality  in  several  articles 
in  Rust,  R.  T.,  &  Oliver,  R.  L.,  eds.  (1994).  Service  quality:  New  directions  in 
theory  and  practice.  Thousand  Oaks,  CA:  Sage  Publications. 

10.  FAR  16.404(a)  and  16.405-2(b)  say  that  award  fee  incentives  are  to  be  used 
only  when  it  is  not  possible  develop  objective  incentive  criteria.  Their  use  in 
performance-based  contracts  is  inconsistent  with  the  PBSA  requirement  for 
objective,  measurable  performance  standards. 

1 1 .  “The  feasibility  requirement  in  contracting  for  results  is  that  the  product  must 
lend  itself  to  clear  definition.  Whether  he  is  contemplating  a  fixed  (price)  or  a 
performance  contract  (a  contract  with  incentives),  the  buyer  must  be  able  to 
specify  the  desired  results  in  simple,  straightforward  terms  to  a  prospective  seller. 
These  terms  must  also  be  meaningful  to  a  knowledgeable  third  party  so  that,  if 

a  dispute  arises,  he  can  determine  whether  the  contract  terms  have  been  fulfilled 
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or  not.  In  purchasing  books  or  equipment  or  even  buildings,  the  school  is  usually 
able  to  describe  exactly  the  product  it  is  after.  Such  procurements  as  the  purchase 
of  administrative  services  is  not  so  easy.”  In  Stucker,  J.,  &  Hall,  G.  (1971).  The 
performance  contracting  concept  in  education.  Santa  Monica,  CA:  Rand,  p.  6. 
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CUSTOMER  FOCUS  AND 
ARMY  PROCUREMENT: 
IS  IT  POSSIBLE ? 


Keith  R.  Shelton  and  Dr.  Drumm  McNaughton 


Current  business  scholars  consider  customer  focus  critical  to  competitive 
advantage.  The  literature  is  full  of  research  and  recommendations  considering 
the  what  and  how  of  customer  focus.  Modern  defense  product  developers,  like  all 
modern  business  enterprise,  seek  competitive  advantage.  Customer  focus,  and 
the  promise  of  competitive  advantage  within  that  concept,  is  seen  as  a  critical 
component  of  a  modern  defense  company's  strategy.  This  article  explores  the 
difficulty  of  developing  true  customer  focus  within  the  rather  strict  and  regulated 
Army  procurement  system. 


Over  the  past  few  years,  customer  focus  research  has  occupied  much  space  in 
the  business  literature.  Research  into  the  hows  and  whys  of  customer  focus 
is  conducted  and  reported  and  metadata  studies  pulling  research  together 
are  completed.  Customer  focus,  we  are  told,  holds  the  key  to  modem  competitive 
advantage. 

Of  course,  customer  focus  is  not  altogether  new.  Strong  and  Harris  (2004)  point 
out  that  since  the  1950s,  marketers  have  “advocated  the  pursuit,  development,  and 
maintenance  of  a  customer-oriented  company”  (p.  1).  Boulding,  Staelin,  Ehret,  and 
Johnson  (2005)  point  out  that  as  far  back  as  1960,  Levitt  counseled  focusing  on 
filling  customer  needs  rather  than  simply  selling  products.  This  idea  that  customer 
focus  is  central  to  competitive  advantage  has  almost  become  the  “leitmotif  for  all 
organizations”  (Strong  &  Harris,  p.  1). 

While  we  accept  customer  focus  as  important  to  corporate  success,  figuring  out 
how  to  be  customer  focused  is  not  altogether  clear,  especially  for  a  procurement 
system  as  complex  as  that  of  the  defense  industry.  The  defense  industry,  long  a 
product-based  industry,  is  attempting  to  capture  the  power  of  customer  focus.  This 
is  problematic  not  only  because  of  its  bias  toward  the  push/pull  marketing  inherit  in 
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product-based  companies,  but  also  because  of  the  complexity  of  the  whole  defense 
procurement  process.  Within  this  process,  even  the  definition  of  customer  is  difficult, 
and  maximizing  customer  needs,  wants,  and  desires  is  more  difficult  still.  This 
article  documents  the  difficulty  defense  contractors  face  in  developing  and  executing 
customer  focus  and  puts  forward  a  plan  for  making  such  a  change. 


CUSTOMER  FOCUS:  WHAT  IS  IT? 

The  first  step  in  setting  up  a  customer  focus  in  defense  contracting  is  to  define  what 
we  mean  by  customer  focus.  The  definition  is  not  altogether  clear  and  runs  the  gamut 
from  simple  data  collection  to  a  rather  complicated  and  amorphous  idea  of  shaping 
customer  experiences. 

CUSTOMER  FOCUS:  WHY  IT  IS  IMPORTANT 

Before  customer  focus  is  defined,  perhaps  an  explanation  of  why  customer  focus 
came  to  be  so  important  is  in  order.  Urban  (2005)  shows  how  customer  power  has 
developed  over  the  past  few  years.  This  market  power  has  developed  as  a  result  of  the 
abundance  of  easily  obtained  information  available  to  consumers.  Urban  points  out 
that  the  Internet  provides  huge  amounts  of  data  that  customers  mine  to: 

■  compare  products, 

■  find  competing  products, 

■  buy  from  suppliers  all  over  the  world,  and 

■  collaborate  with  each  other  concerning  how  one  company  compares  to  others  in 
customer  service. 

This  increase  in  customer  knowledge  has  led,  as  Urban  notes,  to  considerable 
power  shifting  from  the  producer  to  the  customer.  Those  familiar  with  defense 
procurement  understand  that  this  power,  only  now  moving  to  the  customer  in  the 
commercial  world,  has  long  resided  with  the  customer  in  the  defense  procurement 
world. 

CUSTOMER  FOCUS:  A  DEFINITION 

Perhaps  the  simplest  definition  of  customer  focus  can  be  deduced  from  the 
Customer  Relationship  Management  (CRM)  movement.  The  CRM  advocates 
believe  that  the  key  to  customer  focus  is  for  the  firm  to  form  a  “relationship”  with 
the  customer  (Boulding  et  al.,  2005;  Jayachandran,  Shurma,  Kaufman,  and  Raman, 
2005;  Urban,  2005).  Amin  (2005)  puts  the  idea  of  CRM  most  succinctly:  “the  only 
way  for  a  business  to  prosper  in  today’s  highly  competitive  marketplaces  is  through 
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concentrating  passionately  and  devotedly  on  customers  and  their  needs”  (p.  1). 

The  point  of  this  concentration  is  the  formation  of  a  relationship  between  customer 
and  supplier  that  will  hopefully  lead  to  gaining  new  customers  and  retaining  old 
customers,  thus  positively  influencing  (in  the  company’s  direction)  customer  buying 
habits.  Once  that  relationship  is  formed,  it  must  be  managed  (Boulding,  et  al.,  2005). 

At  its  foundation,  CRM  demands  the  accumulation  of  a  tremendous  amount  of 
data.  According  to  CRM,  a  relationship  with  the  customer  cannot  be  established 
unless  the  firm  knows  and  understands  the  customer.  This  knowledge  and 
understanding  begins  with  the  accumulation  of  data. 

CUSTOMER  FOCUS:  ITS  INTENTION 

But  what  is  the  intention  of  customer  focus?  It  is  well  and  good  to  speak  of 
the  customer-firm  relationship  and  the  information  that  can  be  obtained  from  the 
mountain  of  data  that  is  generated  each  year  by  modern  companies,  but  there  must 
be  a  point  to  the  whole  process  more  than  just  getting  to  know  one’s  customers. 
Johnson  and  Seines  (2004)  articulate  this  purpose  when  they  talk  about  building 
“value  for  a  firm  across  an  entire  portfolio  of  customer  relationships”  (p.  2).  Payne 
and  Frow  (2005)  talk  about  “improved  shareholder  value  through  the  development 
of  appropriate  relationships  (p.  2).  Johnson  and  Seines  (2004)  mention  a  host  of 
studies  that  show  a  strong  correlation  between  “the  customer  relation  orientation  and 
its  financial  and  marketing  performance.”  Boulding,  et  al.  (2005)  soften  the  idea  of 
increased  shareholder  value  a  bit  when  they  speak  of  creating  value  for  both  the  firm 
and  the  customer,  but  the  point  of  customer  focus  is  and  has  always  been  the  creation 
of  value  for  the  firm.  While  in  a  market  system  value  creation  (read  profit)  is  not  bad, 
profit  within  the  context  of  defense  procurement  is  somewhat  problematic. 

Customer  focus  insists  that  data  collection  eventually  leads  to  relationship.  But 
what  does  relationship  mean?  Almost  universally,  relationship  means  understanding 
the  customer  based  on  knowledge  of  the  customer  (Gulati  &  Oldroyd,  2005; 
Jayachandran,  et  al.,  2005;  Boulding,  et  al.,  2005).  The  important  thing  is  what 
the  firm  does  with  this  understanding.  Authors  report  many  different  uses  for  this 
understanding.  Gulati  and  Oldroyd  note  that  some  companies  use  this  information 
simply  to  predict  future  customer  behavior  for  the  purpose  of  more  efficient 
marketing  activities  (doing  what  companies  have  always  done,  only  being  more 
efficient  at  it).  Jayachandran,  et  al.  find  many  companies  who  simply  wish  to 
distinguish  profitable  from  unprofitable  (or  less  profitable)  customers.  Others  desire 
developing  new  and/or  different  products  to  fill  customer  needs  (Boulding,  et  al.; 
Johnson  &  Seines,  2004)  or  the  ability  to  match  customers  to  existing  products 
(Johnson  &  Seines).  Some  companies  use  this  understanding  as  a  way  of  developing 
customer  loyalty  (Gulati  &  Oldroyd)  and  developing  long-term  relationships 
(Boulding,  et  al.). 

Others  see  a  less  concrete,  almost  ethereal  purpose  for  this  understanding.  Prahalad 
and  Ramaswamy  (2000)  believe  that  this  understanding  can  allow  the  firm  and  the 
customer  to  “co-create  personalized  experiences”  (p.  4),  Urban  (2005)  sees  the  firm 
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becoming  the  customers’  “advocate,”  and  Johnson  and  Seines  (2004)  want  customers 
to  become  “partners”  with  the  firm.  But  whatever  the  intent  of  the  relationship,  the 
end  result  is  always  seen  as  building  value  for  the  firm. 

CUSTOMER  FOCUS:  HOW  TO  OBTAIN  IT 

Once  the  company  defines  customer  focus,  the  question  becomes,  “How  do  I 
get  it?”  Generally  writers  see  this  as  a  process  moving  from  an  internal  focus  to  a 
customer  focus.  Gulati  and  Oldroyd  (2005)  see  four  stages  as  companies  move  toward 
customer  focus.  These  four  stages  include: 

1 .  gathering  information, 

2.  gaining  insight  from  past  customer  behavior, 

3.  understanding  of  possible  future  behavior,  and 

4.  real-time  response  to  customer  need. 

Johnson  and  Seines  (2004,  p.  2)  suggest  that  the  firm  move  from  “strangers  to 
acquaintances,”  then  from  “acquaintances  to  friends,”  and  from  “friends  to  partners.” 
Prahalad  and  Ramaswamy  (2000)  state  that  the  firm  must  “engage  their  customers 
in  an  active,  explicit,  and  ongoing  dialogue;  mobilize  communities  of  customers; 
manage  customer  diversity;  and  co-create  personalized  experiences  with  customers” 
(p.  1).  Strong  and  Harris  (2004)  insist  the  firm  must  create  relational  tactics, 
procedural  tactics,  and  human  resources  (HR)  tactics  to  create  effective  customer 
focus.  Urban  (2005)  insists  customer  focus  cannot  be  developed  until  a  deep  level  of 
trust  exists  between  the  customer  and  the  firm. 

While  each  consultant  might  have  a  different  approach  to  gaining  customer  focus, 
one  thing  is  sure:  Customer  focus  is  not  business  as  usual.  It  involves  some  type 
of  special  relationship  between  the  firm  and  the  customer  that  is  systematically 
developed  by  the  firm. 


CUSTOMER  FOCUS  AND  PROBLEMS  IN 
THE  ARMY  PROCUREMENT  SYSTEM 

Anyone  working  in  the  Army  procurement  system  understands  that  DoD  rules, 
regulations,  and  legislation  introduce  certain  problems  that  are  not  faced  in  non¬ 
defense  markets,  which  certainly  inhibit  realization  of  the  full  promise  of  customer 
focus.  Many  of  these  problems  cannot  be  overcome  by  anything  less  that  an  overhaul 
of  the  legal  and  regulatory  foundation  of  the  process.  However,  there  are  problems 
that  currently  inhibit  the  realization  of  a  strong  customer  focus  that  can  be  overcome. 
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THE  PROBLEM:  WHO  IS  THE  CUSTOMER? 

The  first  problem  encountered  with  the  Army  procurement  system  is  determining 
who  the  customer  really  is.  A  quick  answer  to  this  question  is  always  the  user,  the 
combat  soldier.  This  is  the  emotionally  satisfying  answer.  All  defense  contractors 
want  to  provide  products  and  services  to  the  combat  soldier  that  will  allow  them  to 
accomplish  the  mission  and  return  home  safely.  Public  relations  demand  this  sort  of 
“user  as  customer”  focus. 

Unfortunately,  this  cannot  be  the  only  answer.  If  the  ultimate  impact  of  customer 
focus  is  to  influence  the  procurement  of  particular  products  and  enhance  corporate 
profitability,  the  user  cannot  be  the  customer.  In  the  Army  system,  the  users  (the 
combat  soldiers)  do  not  control  any  of  the  actual  procurement  processes.  Combat 
soldiers  do  not  control  needs  assessment  and  capability  determination,  concept 
refinement,  contract  award,  product  procurement,  product  development,  or  product 
testing.  While  developing  a  relationship  with  the  combat  soldier  is  emotionally 
satisfying,  it  does  not  produce  the  required  influence  within  the  development  and 
procurement  communities  that  is  the  ultimate  goal  of  a  customer-focused  strategy. 


All  defense  (ontrartors  want  to  provide  produtts  and 
servites  to  the  iombat  soldier  that  will  allow  them  to 
aaomplish  the  mission  and  return  home  safely. 


If  the  combat  soldiers  are  not  the  customers,  then  who  is?  This  answer  depends  on 
where  a  particular  product  is  in  the  product  life  cycle.  Early  in  the  development  cycle, 
the  Joint  Capability  Integration  Development  System  (JCIDS)  community  is  the 
customer  (Chairman  of  the  Joint  Chiefs  of  Staff,  2005).  Later,  the  combat  developer 
(CBTDEV)  community  is  seen  as  the  customer  (PM-2001-ISE,  2003).  Later  still, 
the  material  developer  (MATDEV)  community  is  the  customer  (PM-2001-DL, 

2005).  Throughout  the  whole  process,  the  various  proponent  schools  are  customers. 
Relationships  must  be  formed  with  each  of  these  communities  if  any  influence  is  to 
be  gained. 

For  an  effective  customer  focus  and  its  resultant  procurement  influence,  a  military 
contractor  must  forge  relationships  with  all  relevant  military  stakeholders.  That  might 
mean  relationships  with  the  JCIDS  community  as  they  uncover  capability  gaps; 
relationships  with  relevant  CBTDEVs  as  they  develop  requirements  to  fill  those 
capability  gaps;  and  relationships  with  MATDEVs,  proponent  schools,  and  the  test 
community  as  they  develop,  procure,  and  support  various  products.  This  obviously 
makes  for  a  bewildering  mix  of  relationships,  because  each  of  these  customers  has  a 
different  set  of  needs  and  concerns. 
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THE  PROBLEM:  A  RIGID  PROCESS 

Added  to  the  difficulty  of  determining  the  actual  customer  is  the  inherent  rigidity 
of  the  procurement  process.  Central  to  the  idea  of  customer  focus  is  the  notion  that  a 
company  can  determine  a  need  and  rapidly  fill  that  need  (Gualati  &  Oldroyd,  2005). 
However,  the  Army  procurement  process  is  painfully  slow.  Each  stage  in  this  process 
is  tightly  controlled  by  statute  and  regulation,  such  that  a  defense  contractor  cannot 
quickly  capitalize  on  an  innovation  or  a  technological  breakthrough  by  marketing  and 
selling  a  new  product  to  combat  soldiers. 

Every  piece  of  Army  hardware  developed  and  fielded  to  soldiers  goes  through 
a  very  structured  process.  The  process  has  seven  stages,  beginning  with  a  needs 
and  capability  assessment  phase  and  a  concept  refinement  phase.  Once  through  the 
concept  refinement  phase,  the  future  product  enters  a  technology  development  phase 
and  a  system  development  and  demonstration  phase  in  which  a  product  is  designed 
and  developed.  After  successfully  completing  its  product  demonstration  phase,  an 
Army  product  enters  the  production  and  deployment  phase  where  the  product  is 
manufactured  and  fielded  to  tactical  units.  The  longest  phase  for  Army  products  is 
the  operations  and  support  phase.  This  phase  can  run  in  excess  of  20  years.  Finally, 
the  product  is  removed  from  service  in  the  disposal  phase.  The  Army  tightly  controls 
each  of  these  stages. 

THE  PROBLEM:  PUBLIC  RELATIONS 

Additionally,  there  is  the  basic  public  relations  problem  of  a  corporation  making  a 
profit  at  the  expense  of  combat  soldiers  who  desperately  need  reliable  and  effective 
products  to  do  their  jobs.  While  everyone  acknowledges  the  right  of  defense 
contractors  to  make  a  fair  profit,  the  definition  of fair  is  continually  discussed, 
and  the  MATDEVs  go  to  great  lengths  to  monitor  the  profit  made  by  contractors. 
Furthermore,  the  Army  insists  that  there  be  competition  in  all  areas  of  hardware 
procurement.  The  Army  actively  supports  multiple  suppliers  for  each  product  type  or 
technology  and  goes  to  great  lengths  to  maintain  this  competition  among  contractors 
(Department  of  the  Army,  1989).  Within  the  Army  procurement  process,  gaining 
competitive  advantage  is  difficult,  and  maintaining  that  advantage  is  more  difficult 
still. 

THE  PROBLEM:  PRODUCT  FOCUS 

Complicating  things  further  is  the  fact  that  most  major  military  contractors  are 
organized  with  a  product- focused  structure.  The  military  products  industry  has  four 
major  players:  Boeing,  Lockheed  Martin,  Northrop  Grumman,  and  Raytheon.  Of 
these,  only  Boeing  has  attempted  any  sort  of  customer  focus  (Boeing,  2005),  but 
closer  examination  reveals  that  even  Boeing,  within  its  “customer  faced”  business 
units,  reverts  to  product  focus.  Such  a  product  focus  tends  to  force  a  company  to  look 
inward  rather  than  outward  toward  the  customer.  Consequently,  the  tendency  is  to 
make  really  high-tech,  reliable  new  products  and  force  those  products  to  fit  existing 
Army  needs. 
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This  tendency  to  look  inward  can  be  seen  in  the  vision  statements  of  the  big  four 
defense  contractors  as  found  in  their  Web  sites.  In  each  of  the  vision  statements, 
the  focus  is  internal.  The  vision  has  to  do  with  corporate  success.  It  is  telling  that  in 
each  of  the  corporate  Web  sites  detailing  the  company  vision,  customer  satisfaction 
typically  shows  up  as  a  strategic  thrust  or  value  statement  (Boeing,  2005;  Lockheed 
Martin,  2005;  Northrop  Grumman,  2005;  Raytheon,  2005),  but  the  vision  statement 
and  the  strategies  created  tend  to  revolve  around  products,  not  customers.  This 
internally  focused,  product-related  structure  serves  as  a  strong  barrier  to  developing 
an  outward-looking  customer  focus. 

The  product  focus  is  a  natural  outgrowth  of  technology-based  companies. 
Companies  like  Boeing,  Lockheed  Martin,  Northrop  Grumman,  and  Raytheon 
employ  engineers,  technologists,  and  scientists  whose  education  and  training  tends 
to  be  focused  on  using  technology  to  invent  things :  products.  Their  operations  and 
manufacturing  organizations  are  focused  on  producing  things :  products.  It  is  natural 
that  a  product  focus  emerges. 


WHAT  IS  POSSIBLE 

Thus  far  this  article  has  explored  what  customer  focus  is  and  the  difficulties  of 
influencing  Army  procurement.  Now  the  focus  looks  at  what  is  possible  within  the 
Army  procurement  system.  There  are  several  avenues  that  could  be  explored.  First,  if 
defense  contractors  focus  on  the  dominant  customer  at  each  stage  in  the  procurement 
process,  they  can  use  their  vast  technological  knowledge  and  experience  to  influence 
how  capability  gaps  are  defined  and  how  they  can  best  be  filled.  Additionally,  they 
can  influence  how  spiral  and  technology  upgrades  are  inserted  into  systems  already 
fielded.  If  a  contractor  can  forge  relationships  with  all  procurement  customers  and 
establish  a  sense  of  trust  with  those  customers,  competitive  advantage  through 
procurement  influencing  could  be  achieved,  at  least  at  some  level.  The  key  is  for  the 
customer  to  develop  a  sense  of  trust  in  the  company  (Boulding,  et  al.,  2005;  Johnson 
&  Seines,  2004;  Urban,  2005)  that  leads  to  customer  commitment  (Johnson  &  Seines, 
2004). 

Second  and  less  obvious,  a  defense  contractor  can  use  a  customer-focused  strategy 
to  leverage  productivity  and  enthusiasm  within  the  firm  itself.  Defense  contractors 
tend  to  be  very  patriotic.  They  tend  to  respond  to  soldiers,  especially  combat  soldiers. 
When  contractors  hear  good  words  from  soldiers  about  their  products,  they  feel  great 
pride.  When  they  hear  that  soldiers  are  having  problems  with  the  hardware,  they 
tend  to  be  depressed.  Defense  contractors  want  to  be  a  positive  force  in  the  nation’s 
defense.  Companies  can  use  this  patriotism  by  focusing  on  the  customer  to  build 
morale,  productivity,  and  great  enthusiasm. 

A  third  area  where  customer  focus  might  provide  some  benefit  is  in  the  area  of 
branding  and  brand  recognition.  It  is  extremely  difficult  to  build  brand  identity  with 
defense  products.  Corporate  names  do  not  typically  appear  on  warfighter  products, 
and  everything  the  Army  uses  is  painted  olive  drab,  white,  or  black.  But  if  a  company 
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can  build  a  strong  customer  focus,  build  strong  trust  among  the  various  customer 
groups,  and  build  solid  relationships,  positive  branding  can  be  achieved,  at  least 
within  a  particular  customer  community. 


HOW  TO  PROCEED 

If  procurement  influence  is  possible,  at  least  at  some  level,  through  customer  focus, 
we  must  now  turn  our  attention  to  how  that  influence  can  be  attained.  There  are  three 
avenues  that  shall  be  explored:  getting  to  know  the  customer,  aligning  the  firm,  and 
developing  trust. 

HOWTO  PROCEED:  KNOW  YOUR  CUSTOMER 

Virtually  all  writers  on  customer  focus  understand  that  the  beginning  place  for 
developing  an  adequate  customer  focus  is  to  know  and  understand  the  customer. 

This  knowledge  and  understanding  starts  with  the  accumulation  of  information  about 
the  customer.  This  information  gathering  isn’t  a  one-time  activity.  It  is  a  continuous 
enterprise  that  can  take  years  (Gulati  &  Oldroyd,  2005).  Jayachadran,  et  al.  (2005) 
emphasizes  the  importance  of  the  proper  processing  of  the  mountain  of  data 
accumulated.  Without  adequate  information  processing,  the  customer  focus  effort  can 
be  “rendered  ineffective  by  poor  communication,  information  loss  and  overload  and 
inappropriate  information  use”  (Jayachadran,  et  al.,  p.  128). 

This  information  processing  starts  with  focusing  on  and  understanding  the 
customers’  agenda  (Amin,  2005)  and  treating  customers  as  equals  (Prahalad  & 
Ramaswamy,  2000).  It  means  learning  everything  there  is  to  know  about  the  customer 
at  the  lowest  possible  level,  gathering  and  pooling  this  information  from  all  possible 
sources,  and  then  using  that  information  to  construct  a  comprehensive  picture  of  the 
customer — past,  present,  and  future  (Gulati  &  Oldroyd,  2005).  Once  this  information 
is  accumulated,  Prahalad  and  Ramaswamy  suggest  that  the  company  engage  the 
customer  in  an  active  dialogue.  This  dialogue  allows  the  firm  to  properly  process  all 
the  information.  It  also  serves  to  keep  the  customer  interested,  involved,  and  active  in 
the  process. 

HOWTO  PROCEED:  ALIGN  THE  COMPANY 

The  next  thing  the  company  must  do  to  create  a  vibrant  and  effective  customer 
focus  is  to  ensure  the  company  is  properly  aligned.  Proper  alignment  assures  the 
whole  company  is  pulling  in  the  right  direction. 

Proper  corporate  alignment  begins  with  corporate  vision.  Collins  and  Porras 
(1996)  point  out  that  in  a  dynamic  business  environment  where  truly  great  companies 
are  able  to  renew  themselves,  “vision  provides  guidance  about  what  core  to  preserve 
and  what  future  to  stimulate  progress  toward.”  The  vision  must  face  outward  toward 
the  customer  and  call  the  entire  enterprise  to  focus  on  the  needs  of  the  customer. 
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For  modem  American  business,  the  power  of  the  vision  comes  from  the  excitement 
generated  when  people  feel  they  are  part  of  something  greater  than  themselves 
(Flamel  &  Prahalad,  1989)  and  taps  into  values  people  hold  and  believe  to  be 
important  (Bruhn,  2001).  A  corporate  vision  that  focuses  on  the  customer  can  supply 
this  excitement. 

After  the  vision  is  developed,  strategies  that  support  that  vision  must  be  developed. 
Yukl  (2002)  defines  strategy  as  “a  plan  or  blueprint  for  carrying  out  the  mission  and 
attaining  strategic  objectives”  (p.  360).  Without  a  blueprint,  the  organization  cannot 
build  toward  the  vision.  The  strategies  that  are  developed  describe  the  method  by 
which  the  vision  will  be  achieved.  It  is  beyond  the  scope  of  this  article  to  suggest 
specific  strategies  for  defense  contractors;  however,  it  is  imperative  for  defense 
contractors  to  ensure  that  there  is  proper  alignment  of  vision  and  strategy  (Yukl, 

2002;  Kaplan  &  Norton,  2005)  and  that  the  strategy  be  communicated  effectively 
throughout  the  organization.  Kaplan  and  Norton  believe  that  upwards  of  95  percent  of 
the  employees  of  any  firm  do  not  know  or  do  not  understand  the  company’s  strategy. 
They  believe  that  management  spends  too  little  time  in  “strategy  management” 

(p.  2).  The  result  is  a  misaligned  company:  strategy  pointing  one  way  with  employees 
heading  another. 


Defense  contractors  must  develop  dynamic  customer-focused 
visions  and  dynamic  customer-focused  strategies. 


Finally,  the  strategies  employed  by  the  firm  must  be  continuously  evaluated  and, 
when  necessary,  modified.  Kaplan  and  Norton  (2005)  estimate  that  85  percent  of 
executive  leadership  teams  spend  less  than  one  hour  per  month  reviewing  the  firm’s 
strategy.  Without  effective  reviews,  modifications,  and  corrections,  ineffective 
strategies  are  not  replaced. 

Defense  contractors  must  develop  dynamic  customer-focused  visions  and  dynamic 
customer-focused  strategies.  Then  they  must  drive  those  strategies  down  to  the  lowest 
levels  of  the  organization  and  make  subtle  corrections  whenever  necessary  as  the 
vision  takes  hold. 

It  is  not  being  suggested  here  that  the  whole  product-focused  structure  of  the 
organization  should  be  scrapped  and  a  brand  new  customer-focused  structure  be 
developed.  Technology  companies  tend  to  have,  as  discussed  above,  product  biases 
for  some  very  good  reasons.  Attempting  to  eliminate  these  product  biases  would  be 
very  difficult  and  would  probably  be  detrimental  to  the  firm.  What  is  being  suggested 
is  that  changes  could  be  made  that  would  keep  the  strengths  of  the  product  focus  and 
yet  allow  for  a  strong  customer  focus. 
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Govindarajan  (2005)  examines  the  problems  of  developing  new  organizational 
structures  in  response  to  technological  innovation.  He  states  that  there  are  things  the 
new  structure  should  bring  forward  from  the  old  structure  and  things  that  should 
be  left  behind.  The  trick,  obviously,  is  determining  what  to  bring  forward  and  what 
to  leave  behind.  Dean  (2000)  suggests  that  a  front/back  structure  is  the  best  way 
to  provide  for  customer  needs.  Dean  believes  that  a  front  end  that  concentrates  on 
serving  customers  (determining  needs  and  developing  often  complex  solutions 
to  meet  those  needs)  and  a  back  end  that  concentrates  on  products  is  the  best 
organization  for  meeting  customer  needs.  This  front/back  structure  would  allow  for 
retaining  the  best  of  what  Govindarajan  calls  organizational  DNA. 

HOW  TO  PROCEED:  DEVELOP  TRUST 

Finally,  a  look  at  the  last  step  in  the  process:  using  the  information  and  the 
customer-focused,  properly  aligned  firm  to  build  a  relationship  with  the  customer. 
This  relationship  is  most  often  described  in  terms  of  trust  (Boulding,  et  al.,  2005; 
Urban,  2005;  Johnson  &  Seines,  2004).  The  idea  of  trust  seems  to  revolve  around  the 
idea  of  confidence  customers  feel  when  they  believe  their  suppliers  will  treat  them 
fairly  (Boulding,  et  al.).  Boulding,  et  al.  (p.  8)  notes  that  the  collection  of  data  does 
allow  for  the  “differential  treatment  of  customers.” 


Customers  evaluate  the  treatment  rereived  from 
their  suppliers  and  respond  based  on  the  sense  of  trust 
they  have  in  the  firm. 


Customers  evaluate  the  treatment  received  from  their  suppliers  and  respond  based 
on  the  sense  of  trust  they  have  in  the  firm.  That  is  why  Strong  and  Harris  (2004)  stress 
the  importance  of  making  and  keeping  promises.  They  emphasize  the  development 
of  adequate  measures  to  ensure  that  all  commitments  are  met.  In  this  regard.  Urban 
(2005)  suggests  that  all  communications  remain  open  and  honest.  This  means  giving 
the  customer  information  that  might  actually  lead  the  customer  to  a  competitor’s 
product.  Urban  believes  this  type  of  open  communication  develops  the  trust  required 
to  build  lasting  relationships. 

But  what  is  the  purpose  of  this  trust  once  it  is  built?  Writers  are  united  on  the 
intended  result  of  this  developed  trust.  Johnson  and  Seines  (2004)  believe  that  the 
“creation  of  trust  leads  to  the  creation  of  commitment — trust  breeds  trust,  which 
ultimately  increases  commitment  and  results  in  a  shift  from  short-term  exchanges  to 
long-term  relationships”  (p.  3).  They  point  out  that  long-term  relationships  reduce 
the  customer’s  need  to  solve  problems  in  the  traditional  sense  of  “finding  a  better 
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alternative.”  Urban  (2005)  assumes  that  if  the  company  becomes  an  advocate  for 
the  customer,  those  customers  will  reciprocate  with  trust,  purchases,  and  enduring 
loyalty.  In  other  words,  “a  company  advocates  for  the  customer  interests  and 
customers  advocate  for  the  company  by  buying  its  products”  (Urban,  p.  5). 


CONCLUSION 

This  article  has  looked  at  customer  focus  within  the  Army  procurement  system. 
Certainly  there  is  sufficient  theoretical  data  to  show  that  customer  focus  is  a  valuable 
strategy  for  modern  business,  but  there  are  many  reasons  why  a  customer  focus 
would  be  difficult  within  the  Army  procurement  process.  However,  there  are  sufficient 
potential  benefits  to  developing  a  customer  focus  to  make  it  worthwhile.  American 
defense  contractors  will  never,  short  of  a  radical  overhaul  of  defense  procurement 
rules,  regulations,  and  legislation,  be  able  to  deliver  Prahalad  and  Ramaswamy’s 
(2000)  “personalized  experiences”  or  Urban’s  (2005)  “real  time  solution  to  identified 
problems”  as  a  result  of  a  change  to  customer  focus.  But  by  developing  a  deep 
understanding  of  the  customer;  creating  visions,  strategies,  and  structures  that  allow 
the  firm  to  focus  on  customers  wherever  they  are  found  in  the  procurement  process; 
and  working  hard  to  develop  a  bond  of  trust  between  contractor  and  customer,  some 
level  of  influence  can  be  gained.  And  constant  focus  on  the  end  user — the  combat 
soldier — can  build  morale  and  enthusiasm  within  the  firm  and  a  positive  brand  name 
outside  the  firm. 
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LESSONS  LEARNED 
IN  ACQUISITION 
MANAGEMENT 


Dennis  K.  Van  Gemert  and  Martin  W artenberg 


Many  projects  and  programs  fall  short  of  meeting  their  initial  intended  goals. 
Tracing  these  shortfalls  to  their  common  set  of  root  causes  and  analyzing  these 
root  causes  to  find  common  threads  illustrates  opportunities  for  lessons  learned. 
The  authors  of  this  article  examined  these  common  threads  and  referred  to 
their  professional  experience  in  defense  acquisition  and  academic  backgrounds 
in  project  management  and  systems  engineering  to  address  these  issues  and 
propose  strategies  for  countering  their  ill  effects  on  program  performance. 


Programs  succumb  to  the  same  pitfalls,  regardless  of  lessons  learned 

documentation.  Lessons  learned  must  be  institutionalized  into  the  acquisition 
learning  curve.  Documentation  does  not  ensure  institutionalization.  Why  do  we 
not  learn  from  the  past  and  continue  to  make  the  same  mistakes  on  weapon  system 
after  weapon  system?  Doing  what  has  not  worked  before,  but  doing  it  harder  or  with 
more  process,  usually  produces  the  same  poor  results.  As  aptly  put  in  this  anonymous 
quote: 


If  you  always  do  what  you  always  did,  you  always  get  what  you 
always  got.  If  you  do  not  want  what  you  got,  do  not  do  what  you  did. 
If  you  like  what  you  got,  do  it  again. 
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INSTITUTE  SCOPE  MANAGEMENT  TO  AVOID  SCOPE/ 
REQUIREMENTS  CREEP 

Managing  the  project  scope  is  essential  to  maintaining  cost  and  schedule  target 
dates.  Increasing  the  scope  will  almost  assuredly  increase  cost  and  delay  the  schedule 
(see  Figure  1).  It  is  good  practice  to  follow  the  old  adage,  “It  is  not  a  requirement 
until  someone  is  willing  to  pay  for  it.” 

The  incorporation  of  unfunded  or  under-funded  requirements  leads  to 
uncontrollable  scope  growth.  A  detailed  analysis  of  the  impacts  to  the  cost,  schedule, 
and  technical  baselines  should  be  performed  prior  to  implementing  a  change 
proposal.  Any  potential  adverse  impacts  must  be  documented  in  the  risk  registry 
and  managed  until  either  realized  or  successfully  dispositioned  via  mitigation, 
transference,  avoidance,  or  acceptance  (active  or  passive).  A  signed  change  proposal 
does  not  relieve  the  contractor  from  meeting  cost  and  schedule  constraints. 

Never  use  management  reserve  to  cover  the  cost  of  additional  product  features  and 
functions.  Incorporating  features  and  functions  into  a  product  that  are  not  part  of  the 
contractual  statement  of  work  results  in  resources  being  applied  to  noncontractual 
work,  effectively  putting  the  project  schedule  and  cost  at  risk.  Even  if  the  project 
comes  in  ahead  of  schedule  and  under  budget,  it  should  be  at  the  discretion  of 
senior  acquisition  officials,  not  the  rank-and-file  acquisition  workforce,  to  determine 
whether  the  cost  and  budget  reserve  should  be  used  to  add  functionality  to  the 
product  or  applied  to  another  acquisition  that  may  require  additional  funding. 


FIGURE  1.  THE  TRIPLE  CONSTRAINT  DIAGRAM 
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The  Guide  to  the  Project  Management  Body  of  Knowledge  (PMBOK®)  states: 
“Project  Scope  Management  includes  the  processes  required  to  ensure  that  the 
project  includes  all  the  work  required,  and  only  the  work  required,  to  complete  the 
project  successfully”  (Turner,  1992).  Create  a  scope  management  plan,  perform 
scope  verification,  and  actively  exercise  scope  control,  as  described  in  Chapter  5  of 
the  PMBOK®  Guide.  During  initial  scope  planning,  prioritize  the  triple  constraint 
variables.  For  example,  quality  tends  to  be  an  inflexible  variable,  whereas  availability, 
maintainability,  and  reliability  are  components  of  quality.  Determining  relative 
sensitivities  among  triple  constraint  variables  will  facilitate  system  requirements 
trades  performed  during  critical  points  in  the  program. 

Not  all  requirements  are  equal.  Another  area  requiring  detailed  discussion  and 
documentation  in  requirements  specifications  is  the  relative  priority  of  sets  of 
requirements.  This  allows  the  systems  engineer  and  the  project  engineer  to  make 
concessions,  changes,  and  alternative  approaches  based  on  the  current  design  reality. 


RECOGNIZE  IMMATURE  TECHNOLOGY 

Immature  technology  can  bring  a  program  to  its  knees.  Schedule  elongation  on 
a  research  and  development  (R&D)  project  that  is  composed  almost  entirely  of  the 
technology  development  core  team  is  relatively  inexpensive  compared  to  holding  up 
a  large  program,  burdened  with  sizable  overhead  and  product  teams  unrelated  to  the 
emerging  technology.  When  technology  fails  to  mature  at  the  rate  estimated  during 
initial  planning,  the  costs  of  overhead  and  labor  grow.  Considerations  to  keep  in 
mind: 

■  A  key  technology  should  always  be  thoroughly  evaluated  prior  to  formal  program 
kick-off  to  ensure  adequate  confidence  in  the  technical  maturation  process.  A 
small  team  of  scientists,  engineers,  and  technicians  is  less  expensive  than  a  large 
management  infrastructure  and  the  associated  inefficiencies. 

■  Where  feasible,  allow  for  technological  advances  to  be  spiraled  into  a  product, 
thereby  allowing  the  product  technical  maturity  to  grow  with  the  state-of-the-art,  or 
state-of-the-industry,  whichever  is  the  desired  goal. 

■  The  buyer  must  do  due  diligence  in  determining  the  technical  risk  associated  with 
product  technical  maturity  and  avoid  being  blinded  by  the  wow  factor.  If  the  seller  is 
advertising  technical  capability  for  a  price  that  is  out  of  line  with  the  other  bidders, 
raise  the  flag  and  investigate. 

■  Contracts  should  be  customized  to  dissuade  the  seller  from  over-promising  capability 
to  the  buyer  for  the  sole  purpose  of  winning  the  contract.  A  stepped  procurement 
may  be  the  answer,  with  upgrades  or  more  capable  systems  to  follow  as  technology 
evolves  to  the  point  of  acceptable  risk. 
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AVOID  IMPROPER  USE  OF  MANAGEMENT  RESERVE 

Management  reserve  is  a  budget  reserve  set  aside  to  address  the  unknown- 
unknowns  (unk-unks)  on  a  project  or  program.  The  probability  and  impact  of  these 
risks  are  not  only  unknown,  even  the  presence  of  the  potential  risks  may  be  unknown. 
Management  reserve  is  money  set  aside  to  address  such  unexpected  emergencies. 

Often,  this  money  is  used  to  add  scope  to  the  program  in  order  to  keep  the 
customer  happy.  It  is  easy  to  fall  into  the  trap  of  conceding  to  unfunded  scope 
modifications.  The  danger  lies  in  the  realization  of  these  unk-unks,  and  not  having 
sufficient  reserve  remaining  to  address  them  without  increasing  project  completion 
costs. 

Another  purpose  of  the  management  reserve  is  to  guard  against  the  statistical 
likelihood  that  some  people  overestimate  while  others  underestimate  the  budget. 

A  key  input  to  building  selected  reserves  is  the  individual  uncertainty  and  risks 
associated  with  specific  project  tasks.  The  management  reserve  should  be  used 
strictly  for  the  purpose  of  addressing  unforeseen  obstacles  to  program  success,  never 
to  add  additional  scope. 


ENSURE  ADEQUATE  SCHEDULE  RISK  ANALYSIS 

The  Critical  Path  Method  (CPM)  is  a  useful  tool  and,  when  combined  with 
Program  Evaluation  and  Review  Technique  (PERT)  analysis,  can  give  good  results 
for  planning  purposes.  However,  as  the  schedule  matures,  one  should  always  go 
back  and  run  a  Monte  Carlo  simulation  against  the  schedule  to  avoid  being  overly 
optimistic — a  documented  side-effect  of  CPM.  David  Hulett  (1996)  characterized 
the  optimistic  bias  of  CPM  in  his  paper,  Critical  Path  Method  Scheduling:  Some 
Important  Reservations. 

Ignore  the  assertion  that  completion  dates  are  best  represented  as  single  points  in 
time.  They  are  best  represented  as  a  finite  probability  distribution  with  corresponding 
confidence  intervals.  When  a  date  is  stated,  it  should  be  accompanied  by  a  confidence 
level  (e.g.,  the  task  will  be  completed  on  February  5,  2007,  with  a  confidence  of  70 
percent,  or  2/5/07  [70%]).  Most  scheduling  tools  offer  the  ability  to  perform  Monte 
Carlo  simulation  analysis  to  determine  schedule  confidence  and  risk.  In  addition,  Dr. 
Hulett  recommends  that  the  project  or  program  manager  (PM)  analyze  the  network 
and  look  for  nodes  that  represent  a  high  risk  due  to  excessive  implosion  or  explosion, 
and  consider  adjusting  the  schedule  to  make  the  network  less  sensitive  to  the  effect  of 
statistical  variances  of  parallel  activities.  Merge  bias  is  the  term  used  to  refer  to  the 
elevated  risk  levels  experienced  at  schedule  nodes  where  multiple  paths  converge  into 
a  single  path. 

Though  relatively  new  to  project  management,  the  concept  of  Critical  Chain 
Project  Management  (CCPM)  addresses  the  risk  of  multitasking  on  schedule 
visibility  and  viability  based  on  sharing  critical  path  resources. 
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IMPLEMENT  EFFECTIVE  COMMUNICATIONS 

Communication  is  the  most  important  responsibility  of  the  PM.  Ninety  percent 
of  a  PM’s  time  will  be  spent  communicating,  which  facilitates  collaboration  and 
reduces  inefficiencies.  Concurrent  engineering,  collaborative  design,  systems 
integration,  systems  engineering,  and  high  performance  work  teams  depend  on  fluid 
communication  between  teams,  individuals,  contractors,  and  customers/clients.  It  is 
not  merely  enough  that  communication  take  place,  but  that  the  communication  be 
effective.  Be  aware  of  the  barriers  and  facilitators  of  communication.  Use  clear  and 
concise  communication,  as  well  as  active  listening. 


Those  in  management  must  take  tare  not  to  lose  touth 
with  the  information  grapevine:  peer-to-peer  information 
flow  at  the  design-  and  user-working  level. 


Management  must  actively  engage  the  engineering  workforce  and  the  user 
community  and  seek  their  input  and  advice.  Those  in  management  must  take  care 
not  to  lose  touch  with  the  information  grapevine:  peer-to-peer  information  flow  at 
the  design-  and  user- working  level.  Even  those  with  the  best  of  intentions  to  stay 
connected  to  their  former  colleagues  must  take  note  that  a  managerial  position 
elevates  an  individual  above  the  day-to-day,  word-of-mouth  conversation  about  what 
is  working  and  what  is  not.  Those  actively  engaged  in  solving  the  design  issues  have 
first-hand  information  about  what  is  and  is  not  working,  so  it  is  advisable  to  pay 
attention  to  their  concerns.  Often,  this  communication  flow  is  hindered  by  a  belief 
that  management  knows  better  then  those  in  the  trenches  what  is  and  is  not  working. 
And  often,  when  input  is  sought  from  management,  the  response  is  not  taken  to  task. 

Do  not  be  afraid  of  bad  news,  and  never  shoot  the  messenger,  lest  the  free  flow 
of  communication  be  inhibited.  Better  to  learn  of  bad  news  while  there  is  still  time 
to  correct  or  mitigate  it  than  when  it  is  too  late  to  react.  Never  conceal  unfavorable 
information  about  project  or  program  progress.  Analyze  the  threat  and  probability 
levels,  and  search  for  solutions.  By  doing  so,  when  the  sponsor  is  briefed,  you  have 
shown  that  there  is  a  negative  issue  or  risk,  but  that  it  is  being  actively  addressed. 
Always  give  the  sponsor  an  opportunity  to  participate  in  the  solution.  It  is,  after  all, 
the  sponsor’s  product.  While  industry  has  become  diligent  in  documenting  lessons 
learned,  communicating  that  information  has  not  been  as  successful.  Fear  of  reprisal 
is  the  leading  cause  of  bad  news  not  flowing  up  the  chain  of  command. 
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REDUCE  PROGRAM  OFFICE  ROTATIONS 

Continuity  of  leadership  is  essential  to  effective  and  efficient  program  execution. 
When  program  officers  are  rotated  out  every  few  years,  the  learning  curve  recycles 
before  utility  of  the  previous  learning  cycle  can  be  harnessed.  In  effect,  once  an 
officer  gains  the  background  understanding  of  the  program,  keep  the  individual  in 
place  long  enough  to  realize  their  talent  in  that  position.  Turning  over  leadership 
every  few  years  keeps  the  program  office  in  a  state  of  constant  redirection,  shifting 
priorities  and  expectations,  and  confusion.  It  generally  takes  several  years  on  a 
particular  program  or  project  to  reach  peak  performance. 

Plans  should  be  in  place  for  an  orderly  transfer  of  responsibility  and  knowledge. 
Only  a  small  part  of  project  activities  and  communications  are  in  some  form  explicit. 
Much,  including  off-the-table  agreements,  are  tacit  and  need  to  be  transferred  or 
acknowledged  by  new  folks  coming  into  the  project.  Succession  planning  is  essential 
to  program  or  project  success. 

The  success  of  the  U.S.  Navy  Strategic  Systems  Project  Office  in  the  management 
of  the  Polaris,  Poseidon,  and  Trident  Programs  is  due  at  least  in  part  to  the  fact  that 
key  PMs  stayed  on  the  job  and,  when  rotated,  moved  into  other  parts  of  the  program, 
so  that  key  tacit  knowledge  remained  available.  Aside  from  the  Navy’s  Fleet  Ballistic 
Missile  (FBM)  program  and,  to  a  lesser  extent,  the  F-18  E/F,  rapid  turnover  and  a 
punch-your-ticket  mentality  is  prevalent  in  defense  acquisition.  Despite  the  laudable 
results  achieved  on  the  FBM  and  F-18  E/F  programs,  the  rest  of  the  acquisition/ 
procurement  world  did  not  follow  the  model  the  Systems  Projects  Office  and  the 
Strategic  Systems  Project  Office  developed  regarding  time  in  place  and  rotating 
within  the  program.  There  have  been  studies  and  the  subject  has  been  extensively 
discussed  at  the  Naval  Postgraduate  School  in  Monterey  as  well  as  at  the  Defense 
Systems  Management  College  (DSMC). 


RECOGNIZE  OVERLY  AGGRESSIVE  BIDDING 

Be  cautious  of  over-aggressive  bidding,  and  protect  against  bids  that  are 
unrealistic.  Ensure  that  all  scope  has  been  accounted  for,  and  review  the  scope  against 
cost  for  any  inconsistencies.  Use  the  change  clause  for  protection,  and  incentivize 
the  bidding  contractors  to  be  reasonably  accurate  in  their  estimated  costs.  Share  the 
budget  savings,  as  well  as  the  losses,  with  the  contractors,  if  possible. 

Past  experience  should  serve  as  a  reality  check  on  what  is  achievable  and  what 
is  overly  optimistic.  If  greater  capability  of  a  previous,  similar  system  is  being 
proposed  at  a  cost  that  is  not  proportional  to  that  capability,  seek  justification  for 
this  discontinuity.  The  contractor  may  be  bidding  aggressively  under  competitive 
pressures,  with  the  intent  to  make  up  any  losses  during  the  production  and/or 
operations  and  maintenance  phases.  The  buyer  usually  ends  up  paying  for  cost 
growth,  regardless  of  the  contract  agreement. 
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One  vehicle  for  the  seller  (contractor)  to  recoup  lost  income  is  to  buffer  the 
engineering  change  proposals,  or  contract  change  proposals.  If  the  seller  is  not 
meeting  their  fair  profit  goals,  expect  to  pay  more  for  design  changes  than  if  they  are 
meeting  goals.  This  does  not  mean  that  the  seller  is  trying  to  take  advantage  of  the 
buyer.  The  seller  is  interested  in  making  a  fair  and  reasonable  profit,  while  the  buyer 
is  interested  in  receiving  the  most  capability  for  their  expenditure. 

Keep  procurement  dialogue  open  and  honest  on  both  sides  of  the  contract.  Mistrust 
of  the  other  side  is  not  in  either  side’s  best  interest.  A  realistic  should  cost  model 
must  be  used  to  examine  bids  for  reasonableness.  Bids  that  are  too  low  should  be 
questioned  in  detail  regarding  how  the  organization  plans  to  meet  the  low-cost  targets. 

“In  nature,  the  optimum  is  almost  always  in  the  middle  somewhere. 

Distrust  assertions  that  the  optimum  is  at  an  extreme  point.’’ 

—  Professor  David  Akin,  University  of  Maryland 


ESTABLISH  STAFFING  AND  RESOURCE  PLANNING 

It  is  not  uncommon  for  new  employees  to  wait  months  for  proper  security 
clearances  to  move  through  the  process.  It  is  important  not  only  to  account  for  this 
delay  in  the  staffing  plan,  but  also  to  use  the  time  to  prepare  for  productive  integration 
of  the  new  team  member.  Plan  for  the  individual’s  arrival  by  ensuring  a  computer  and 
other  needed  work  equipment  and  items  are  on  his/her  desk  and  in  working  order. 
Nothing  is  more  frustrating  and  unproductive  than  waiting  several  months  for  a 
security  clearance  to  come  through,  only  to  find  out  that  it  will  take  a  few  more  days 
to  get  a  working  computer  set-up,  and  a  few  weeks  to  order  more  licenses  for  the 
software  tool  required  to  perform  primary  job  responsibilities  (e.g.,  computer-aided 
design  and  analysis  software).  If  you  have  several  months’  notice  of  an  individual’s 
start  date,  take  advantage  of  this  time  to  set  up  the  workspace  and  determine  if  more 
software  licenses  are  required.  This  is  an  easy  money  saver,  but  often  overlooked,  thus 
costing  a  fortune  in  lost  productivity. 


LEVERAGE  SYSTEMS  ENGINEERING  WITH  PROJECT  MANAGEMENT 

The  ability  of  a  project  team  to  successfully  complete  a  project,  and  the  follow- 
on  work  associated  with  it,  is  tied  to  the  team’s  ability  to  leverage  the  capabilities  of 
systems  engineering  and  project  management,  enabling  collaborative  teaming  among 
various  engineering  disciplines.  Leveraging  the  synergies  between  the  two  disciplines 
(as  shown  in  Figure  2)  is  a  critical  aspect  of  collaboration. 

The  two  disciplines  overlap  in  some  respects  and  complement  each  other  in  others. 
Where  they  overlap,  systems  engineering  and  project  management  are  motivated  by 
different  objectives  resulting  from  their  unique  perspectives.  The  PM  is  concerned 
with  maintaining  cost  and  schedule  commitments,  whereas  the  systems  engineering 
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FIGURE  2. 

PROJECT  MANAGEMENT  AND  SYSTEMS  ENGINEERING  RESPONSIBILITIES 


emphasis  is  on  requirements  management  and  verification.  Today’s  technology 
often  results  in  a  family-of-systems  or  system-of-systems  environment  where 
various  technologies  are  connected  through  an  information  grid  in  which  systems 
interoperability  is  a  leading  success  criterion  for  project  success.  Methodical  systems 
engineering  is  essential  to  a  well-integrated  system.  The  PM  and  systems  engineer 
must  work  together  to  achieve  optimal  results. 

It  is  helpful  to  review  the  formal  definitions  of  project  management  and  systems 
engineering,  as  used  by  their  industry  governing  bodies.  The  Project  Management 
Institute  defines  project  management  as: 

The  application  of  knowledge,  skills,  tools,  and  techniques  to 
a  broad  range  of  activities  in  order  to  meet  the  requirements  of 
a  particular  project.  Project  management  is  comprised  of  five 
processes — initiating,  planning,  executing,  controlling,  and 
closing — as  well  as  nine  knowledge  areas.  These  nine  areas  center  on 
management  expertise  in  project  integration,  project  scope,  project 
time,  project  cost,  project  quality,  project  human  resources,  project 
communications,  project  risk  management,  and  project  procurement. 

(PMI,  n.d.) 
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The  International  Council  on  Systems  Engineering  defines  systems  engineering  as: 

An  interdisciplinary  approach  and  means  to  enable  the  realization 
of  successful  systems.  It  focuses  on  defining  customer  needs  and 
required  functionality  early  in  the  development  cycle,  documenting 
requirements,  then  proceeding  with  design  synthesis  and  system 
validation  while  considering  the  complete  problem:  operations, 
performance,  test,  manufacturing,  cost  and  schedule,  training 
and  support,  and  disposal.  Systems  engineering  integrates  all 
the  disciplines  and  specialty  groups  into  a  team  effort  forming 
a  structured  development  process  that  proceeds  from  concept  to 
production  to  operation.  Systems  engineering  considers  both  the 
business  and  the  technical  needs  of  all  customers  with  the  goal  of 
providing  a  quality  product  that  meets  the  user  needs.  (INCDSE, 
ad.) 

In  the  complex  world  of  system-of-systems,  project  managers  have  their 
hands  full  just  managing  the  job.  It  is  the  role  of  the  systems  engineer  to  provide 
the  management  of  the  technical  issues  including  quality,  conformance  with 
requirements,  and  other  targets  of  the  specific  procurement  (flexibility,  adaptability, 
reliability,  etc.). 


ENSURE  FOR  ADEQUATE  PLANNING 

Under  pressure  to  perform,  many  PMs  proceed  to  the  product  development  and 
execution  phase  prematurely,  without  sufficient  attention  to  the  planning  phase.  It  is 
important  to  note  that  upfront,  advance  planning  becomes  more  critical  as  budgets 
become  more  constrained.  Proper  planning  dramatically  reduces  scrap,  rework,  and 
redesign.  A  good  rule-of-thumb  for  planning  is  to  spend  approximately  1 5  to  25 
percent  of  the  overall  budget  on  planning:  planning  for  success. 

However,  it  is  equally  important  to  ensure  that  the  budget  spent  on  planning  is 
value-added.  Structure  must  be  enforced  on  the  planning  phase  to  make  certain  that 
the  planning  is  disciplined  and  documented.  The  PMs  and  systems  engineers  are 
essential  disciplines  to  the  planning  process.  Working  together,  they  can  efficiently 
produce  an  integrated  set  of  project  management  (e.g.,  scope  management  plan, 
risk  management  plan,  quality  management  plan)  and  technical  management  (e.g., 
systems  engineering  plan,  systems  engineering  management  plan,  etc.)  plans. 


RECOGNIZE  SHOULD  COST  VS.  WOULD  COST 

If  one  builds  in  a  should  cost  clause  in  the  contract  or  includes  it  as  part  of  a 
value  engineering  section,  one  has  the  ability  to  propose  alternative  approaches  to 
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the  sponsor  in  order  to  save  both  time  and  money.  It  is  one  of  the  least-used  tools  of 
systems  engineering  in  that  once  the  contract  is  signed  alternative  approaches  tend 
to  cease.  Negotiate  a  percentage  sharing  basis  with  the  sponsor;  a  50/50  sharing 
arrangement  is  a  good  initial  objective. 


UTILIZE  SYSTEM  ARCHITECTURE  STUDIES 

Another  area  that  causes  failure  is  an  upfront  lack  of,  or  non-use  of,  system 
architecture  studies  related  to  topics  like  functionality,  modularization,  placement  of 
risk,  determination  of  design  margins,  etc.  Often,  this  may  be  done  by  the  systems 
engineering  team  and  included  as  a  deliverable,  but  may  never  get  into  the  project 
planning  activity. 


CONCLUSION 

A  root  cause  of  a  failure  is  often  not  sufficient  in  and  of  itself  to  cause  a 
catastrophic  failure.  It  is  when  secondary  variables  are  present,  in  a  specific 
sequence,  that  all  the  factors  align  to  cause  the  worst-case  scenario  to  become  reality. 
For  example,  the  common  thread  between  the  two  shuttle  tragedies  is  the  prior 
observation  and  documentation,  during  multiple  flights,  of  a  technical  design  flaw 
that  had  the  potential  for  causing  a  catastrophic  failure.  If  repeated  O-ring  failures  on 
the  Challenger  led  to  disaster,  then  why,  within  20  years,  was  repeated  tile  damage 
to  a  shuttle’s  heat  shield  from  break-away  insulation  allowed  to  continue,  until 
eventually  a  piece  struck  in  such  a  manner  as  to  cause  catastrophic  damage  to  the 
integrity  of  Columbia’s  heat  shield? 

The  answer  lies  in  the  knowledge  management  of  lessons  learned  of  not  only 
the  root  cause  itself,  but  also  the  associated  factors  and  circumstances.  We  have 
become  very  good  at  documenting  lessons  learned,  but  not  so  disciplined  in  the 
institutionalization  of  those  lessons.  Documenting  lessons  learned  is  only  the 
beginning  of  knowledge  management.  Those  lessons  must  be  socialized  among 
colleagues  to  the  degree  that  they  are  transferred  to  upcoming  generations.  How 
do  lessons  learned  become  generally  accepted  best  practices?  Some  lessons  should 
change  the  foundations  of  our  organizational  culture  permanently.  We  have  mastered 
the  archiving  of  lessons  learned.  Now  we  must  master  their  retrieval,  and  give  them 
life,  not  just  a  life  cycle. 

A  final  quote,  attributed  to  Andy  Grove,  chairman  emeritus  of  Intel:  “Individuals, 
processes  and  organizations  are  perfectly  designed  to  achieve  whatever  results  they 
are  currently  getting,  so  if  you’re  not  happy  with  what  you  are  achieving,  its  time 
to  reconsider  your  assumptions  and  approaches  to  your  process  and  product  design 
methods.” 
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TEST  AND  EVALUATION 
LESSONS  LEARNED 
FROM  THE  FIELD 

Karen  M.  Stadler 


This  article  examines  test  and  evaluation  (T&E)  lessons  learned  from  more  than 
300  students  with  extensive  T&E  field  experience  who  attended  the  Defense 
Acquisition  University  (DAU)  test  and  evaluation  classes  during  FY02-FY05. 

The  T&E  lessons  learned  in  18  categories  were  researched  and  correlated, 
and  findings  in  the  top  five  categories  are  presented.  In  particular,  this  article 
focuses  on  detailed  lessons  learned  in  the  areas  of  test  design  and  execution,  test 
planning,  teamwork  and  communication,  funding,  and  scheduling.  A  compilation 
of  student  (field  practitioner)  comments  and  recommendations  is  presented, 
and  overall  results  are  compared  with  results  from  other  similar  studies  and 
documents. 


As  part  of  the  DAU  Advanced  Test  and  Evaluation  (TST  301)  class,  students 
prepare  and  present  PowerPoint  slideshows  on  various  T&E-related  topics. 
Many  students  prepare  and  present  detailed  briefings  on  their  T&E  lessons 
learned.  The  lessons-leamed  presentations  are  typically  based  on  actual  experiences 
in  planning,  conducting,  analyzing,  and  reporting  test  results.  Students  typically 
have  many  years  of  T&E/acquisition  field  experience,  and  their  presentations  contain 
a  wealth  of  valuable  information,  which  could  help  others  avoid  common  sources 
of  error  when  designing  and  executing  test  events.  The  purpose  of  this  article  is 
to  identify  and  discuss  common  T&E  best  practices  and  lessons  learned,  thereby 
enabling  possible  cost  and  schedule  savings  and  improved  test  results  from  future 
T&E  efforts.  Decision  makers  and  acquisition/program  leadership  can  benefit  from 
this  article  by  better  understanding  the  top  T&E  related  issues,  as  reported  by  field 
practitioners. 

The  T&E  lessons-leamed  data  was  obtained  from  students  who  attended  this 
author’s  TST  301  classes  during  FY02-FY05.  Of  the  393  students,  301  students 
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TABLE  1. 

NUMBER  OF  STUDENTS  BY  ORGANIZATION/SERVICE 


ORGANIZATION/SERVICE 

NUMBER  OF 
STUDENTS 

PERCENT  OF 
STUDENTS 

U.S.  Army 

141 

46.8 

U.S.  Air  Force 

89 

29.6 

U.S.  Navy 

35 

11.6 

Missile  Defense  Agency  (MDA) 

24 

8.0 

Special  Operations 

6 

2.0 

Department  of  Defense  (DoD) 

4 

1.3 

U.S.  Marine  Corps 

2 

0.7 

Industry/Contractor 

0 

0.0 

Total  Number  of  Students 

301 

100.0 

presented  significant  T&E  lessons-leamed  information.  The  students  came  from  all 
four  services  and  DoD  agencies,  as  summarized  in  Table  1 .  The  lessons-leamed  data 
consists  solely  of  student  comments  and  opinions  and  is  based  on  student  knowledge 
and  experience  in  the  T&E  area,  along  with  any  research  conducted  by  individual 
students. 

The  student  data  was  analyzed  for  common  trends,  and  1 8  different  categories, 
covering  all  common  trends,  were  selected.  The  student  briefings  were  then  tabulated, 
to  determine  the  number  of  student  briefings  with  lessons  learned  for  each  of  the 
18  categories.  For  example,  192  of  the  301  total  students  (63.8%  of  the  total)  had 
significant  lessons  learned  in  the  area  of  test  design  and  test  execution.  Note  that  each 
student  briefing  contained  lessons  learned  from  one  or  more  of  the  18  categories.  The 
student  lessons  learned  data  is  presented  in  Table  2. 


MAJOR  FINDINGS 

Of  the  18  categories  of  T&E  lessons  learned  in  Table  2,  this  article  will  further 
discuss  the  top  five  categories  (test  design  and  execution,  test  planning,  teamwork  and 
communication,  funding,  and  schedule).  Note  that  far  more  students  (41%-64%)  had 
lessons  learned  in  the  top  five  categories  than  in  the  bottom  13  categories  (5%-30% 
of  students).  This  article  contains  a  summary  and  detailed  student  comments  and 
recommendations  for  each  of  the  top  five  areas.  This  information  can  benefit  the  T&E 
community  by  providing  detailed  lessons  learned,  which  might  assist  future  T&E 
efforts  and  help  acquisition  leadership  better  understand  the  major  T&E  issues  and 
concerns. 
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TABLE  2. 

NUMBER  OF  STUDENTS  WITH  LESSONS  LEARNED,  IN  EACH  CATEGORY 


LESSONS  LEARNED  CATEGORY 

NUMBER  OF 
STUDENTS 

PERCENT  OF 
STUDENTS 

1.  Test  design,  test  methods,  test  execution,  and 
analysis  methods 

192 

63.8 

2.  Test  planning 

162 

53.8 

3.  Teamwork  and  communication 

141 

46.8 

4.  Funding,  budget,  and  cost 

141 

46.8 

5.  Schedule 

122 

40.5 

6.  Test  infrastructure,  test  tools,  test  articles,  and 
instrumentation 

91 

30.2 

7.  Test  requirements 

91 

30.2 

8.  Safety  and  risk  management 

85 

28.2 

9.  Government  leadership  and  management 
issues  and  organizational  politics 

66 

21.9 

10.  Contractor  issues,  including  contractor  leadership 
and  management  issues 

55 

18.3 

11.  Training  issues 

35 

11.6 

12.  Modeling  and  simulation  (used  in  conjunction 
with  testing) 

34 

11.3 

13.  Interfaces,  interoperability,  and  integration 
issues 

33 

11.0 

14.  “Stuff  happens”  (Murphy’s  law,  weather) 

32 

10.6 

15.  Manpower  issues 

27 

9.0 

16.  Immature  technology  and/or  immature  system 

15 

5.0 

17.  Commercial-off-the-shelf  (COTS)  and 
Non-development  Item  (NDI)  issues 

15 

5.0 

18.  Poor  judgment 

15 

5.0 

Total  Number  of  Students 

301 

100.0 

THE  TOP  FIVE  STUDENT  CATEGORIES 

TEST  DESIGN,  TEST  METHODS,  TEST  EXECUTION,  AND  ANALYSIS 
METHODS  (63.8  PERCENT) 

Test  design,  methods,  execution,  and  analysis  methods  are  unquestionably  among 
the  largest  factors  that  determine  the  success  or  failure  of  test  events.  Students  offered 
the  following  advice  in  this  area: 
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■  Take  the  time  to  develop  a  robust  T &E  strategy  and  to  determine  the  best  (anticipated) 
test  design  and  methods  for  your  situation.  Consult  experts  where  necessary. 
Examine  all  facets  of  T&E  such  as  instrumentation,  data  collection,  analysis 
methods,  test  validity,  test  procedures,  etc.  For  example,  one  might  consider  a  side- 
by-side  comparison  of  the  existing  and  the  new  system.  The  new  system  may  not 
meet  all  the  requirements,  but  may  be  significantly  better  than  the  existing  system. 
Without  the  comparison,  the  true  conclusion  may  be  missed. 

■  The  devil  is  in  the  details.  Student  after  student  indicated  that  the  little  details  (as 
well  as  the  overall  test  design  and  execution)  greatly  affected  the  success  or  failure 
of  their  test  events.  For  example,  one  detail  of  test  planning  might  be  to  consider 
collecting  diagnostic  data,  as  well  as  test  data.  In  case  of  equipment  failure,  the  data 
can  assist  in  determining  the  problem(s). 

■  Understand  the  test  objectives,  including  how  and  why  the  test  requirements  were 
generated.  Know  the  what  and  why  behind  limits  and  guidelines.  Look  at  the 
system  requirements,  don’t  just  accept  them.  Spend  the  time  upfront  with  users  and 
requirements  developers  to  get  the  requirements  well  defined,  especially  where  the 
requirements  don’t  make  sense.  Often,  the  user  may  not  know  what  he/she  wants  and 
why.  Is  the  test  relevant?  Are  the  requirements  realistic?  The  user  will  sometimes 
change  the  requirements,  if  the  need  is  explained  to  them.  Systems  engineers  and 
testers  need  to  work  together. 

■  Include  tests  at  realistic  operating  conditions  and  at  all  comers  of  the  envelope. 
These  tests  should  occur  prior  to  production  or  as  early  as  possible.  Measure  all 
critical  parameters  and  verify  all  requirements.  Review  the  Test  and  Evaluation 
Management  Plan  (TEMP)  and  test  plans  versus  requirements  (e.g.,  was  essential 
data  collected  and  nonessential  data  not  collected?). 

TEST  PLANNING  (53.8  PERCENT) 

Numerous  students  stressed  the  importance  of  thorough  test  planning.  There  are 
things  one  cannot  or  will  not  anticipate;  stuff  happens,  people  make  mistakes.  But 
proper  anticipation  and  thorough  planning  will  result  in  fewer  problems  in  the  long 
run,  including  a  better  chance  of  success  and  remaining  within  cost  and  on  schedule. 
Good  planning  allows  proper  resource  allocation  and  makes  test  execution  far  easier. 
Additional  student  comments  concerning  the  importance  of  test  planning  and  not 
cutting  comers  are  as  follows: 

■  Any  test  plan  should  have  more  than  one  person  to  review  it.  Make  use  of  available 
expertise. 

■  Do  analysis  prior  to  testing  to  try  to  predict  and  anticipate  results.  This  analysis  will 
assist  in  identifying  potential  problems  and  developing  contingency  plans.  It  also 
helps  identify  needed  changes  to  test  plans. 
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■  Plan  for  contingencies,  especially  weather.  Have  an  alternate  test  plan/altemate  test 
points  available  for  each  day  of  testing. 

■  When  planning  a  test  with  others,  start  early.  They  were  already  busy  before  you 
came.  Attempt  to  keep  things  simple.  Questions  should  be  clear  and  to  the  point. 
Follow  their  processes,  if  possible. 

■  Become  an  expert  on  systems  you  are  testing.  Tap  into  subject  matter  expert  (SME) 
knowledge.  Spend  time  with  the  user.  Find  out  the  user’s  priorities,  concerns,  and 
the  reasons  for  these.  Consider  traveling  to  the  factory  or  contractor  facility.  Obtain 
and  study  system  documentation.  Discuss  design  criteria  with  contractors,  SMEs, 
users,  maintainers,  etc.  The  smarter  you  are  about  the  system,  the  better  decisions 
you  will  make,  and  others  will  not  be  able  to  hide  issues  that  need  to  be  brought  out 
into  the  open. 

■  Early  operational  test  agency  (OTA)  involvement  is  critical  to  reducing  OT  risk.  The 
OTA  should/could  assist  in  requirements  and  concepts  of  operations  (CONOPS) 
development,  in  early  identification  of  T&E  related  systems  concerns,  in  providing 
input  aimed  at  conducting  selected  developmental  test  (DT)  events  in  a  more 
operational  environment,  etc. 

■  Plan  for  proper  training,  and  make  the  case  for  extra  training  time,  if  needed.  Training 
usually  results  in  a  better  and  safer  test,  better  information,  and  better  decisions.  It 
almost  always  pays  off  in  the  long  run. 

TEAMWORK,  COMMUNICATION  (46.8  PERCENT) 

Many  students  cited  the  importance  of  teamwork  and  communication  as  critical 
to  minimizing  and/or  preventing  T&E  problems.  Frequent,  open,  and  timely 
communication  (integrated  product  team  [IPT]  meetings  or  otherwise),  along  with 
consulting  with  SMEs  when  needed,  undoubtedly  increases  the  chance  of  program 
success.  Students  had  a  number  of  recommendations  concerning  the  importance  of 
teamwork  and  communication: 

■  Get  support  from  and  educate  all  involved  commands  and  customers.  Early 
involvement  of  all  key  parties  (developmental  and  operational  testers,  evaluators, 
user  representatives,  program  office  personnel,  range  and  safety  personnel,  specialty 
area  experts,  contractors,  etc.)  is  critical.  Resolve  conflicts  early,  if  possible. 
Understand  the  decision  cycles  and  decision  criteria  of  all  involved  organizations. 

■  Consult  experts  for  any  important  matter,  when  lacking  the  necessary  expertise. 
Ask  for  help  when  needed! 
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■  Do  not  punish  subordinates  for  finding  errors  or  performing  necessary  rework. 
Establish  a  healthy  team  environment  where  discussions  are  open  and  candid  and 
where  people  do  not  hide  mistakes  and  problems. 

■  The  IPTs  work  well  when  used  correctly.  They  should  consist  of  qualified  and 
empowered  team  members  from  all  key  organizations  and  stakeholders,  plus  any 
needed  SMEs.  Get  the  best  experts  from  each  organization.  Ensure  expectations 
are  well  understood.  If  more  organizations  have  input  and  more  points-of-view 
are  considered,  more  creative  options  may  be  generated,  which  lead  to  better 
decisions  with  better  buy-in.  There  should  be  consistent,  success-oriented,  proactive 
participation:  open  discussions  with  no  secrets.  Issues  and  concerns  should  be 
raised  and  resolved  early  through  critical  dialogue,  not  just  “group  think.”  Reasoned 
disagreement  should  occur,  with  decisions  based  on  reaching  consensus,  if  possible. 
Ethical  decision  making  is  important,  with  action  items  worked  quickly.  Properly 
functioning  IPTs  can  reduce  confusion  in  an  already  complex  process. 

■  The  T&E  effort  on  joint  programs  is  much  tougher  to  coordinate.  Establish  a  joint 
T&E  working  integrated  process  team  (WfPT),  with  the  best  experts  from  each 
service  and  continuous  “up  the  line”  communications.  Joint  programs  are  more 
challenged  by  rice  bowls  and  politics,  so  communication  is  even  more  critical. 
Leadership  and  joint  processes  need  to  be  established  early.  Goals,  schedules, 
performance  levels,  logistics  issues,  and  CONOPS  are  unique  and  different  for  each 
service.  All  these  issues  need  to  be  worked  out  early. 

■  Test  reports  need  to  clearly  communicate  the  facts.  Write  reports  for  all  audience 
levels  (executive  summary  in  lay  terms  for  executives,  common  technical  terms 
for  managers,  attachments  with  appropriate  technical  jargon  for  engineers  and 
analysts).  Not  all  deficiencies  are  equal;  prioritize  and  sort  boulders  from  gravel, 
based  on  mission  impact.  Whenever  a  deficiency  is  mentioned,  address  impact  and 
ease  of  correction.  Include  charts  and  tables  that  are  easy  to  understand.  Put  the 
test  methodology  in  an  appendix.  Report  all  results,  good  and  bad,  and  document 
the  value-added  of  this  testing.  Report  test  results  with  respect  to  conditions  and 
mechanisms.  Report  bottom  line  results — what  worked  and  what  did  not — and  what 
decision  makers  need  to  know  concerning  the  system’s  mission  capabilities  and 
limitations.  Bad  news  does  not  get  better  with  age;  consider  a  quick-look  report  or 
interim  results  if  decision  makers  need  to  quickly  know  the  results. 

FUNDING,  BUDGET,  AND  COST  (46.8  PERCENT) 

This  group  of  lessons  dealt  with  the  importance  of  adequate  funding  and  the 
negative  effects  of  inadequate  funding  on  T&E  programs.  Student  comments  are  as 
follows: 
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■  With  the  current  DoD  budget  situation,  one  of  the  hardest  hit  areas  is  T&E.  But 
money  and  time  saved  by  cutting  comers  up  front  is  invariably  spent  in  fixing 
problems  later.  Customers  are  unhappy,  and  it  leads  to  increased  cost  in  the  long- 
run. 

■  Because  of  funding  limitations,  testers  commonly  execute  only  a  small  subset  of 
the  test  events  that  should  be  required.  Lack  of  funding  results  in  delayed  testing, 
test  events  that  are  limited  in  scope,  data  or  reports  that  are  not  delivered,  cancelled 
test  events,  and/or  a  nonrobust  test  program  that  fails  to  find  the  critical  problems 
or  issues.  Unforeseen  test  requirements  sometimes  arise  for  which  funding  is 
unavailable. 

■  Seek  to  document  and  educate  management  and  leadership  that  cost  savings  obtained 
by  reduced  testing  may  compromise  or  jeopardize  final  system  quality  or  operation. 
And  reduced  testing,  which  allows  systems  to  be  fielded  with  undetected  problems, 
could  potentially  endanger  lives. 

■  To  mitigate  the  risk  of  inadequate  funding,  seek  to  determine  realistic  cost  estimates 
early  in  program  development.  Conduct  thorough  budget  planning  and  review, 
consult  with  experts  as  needed,  and  include  adequate  management  reserve  for 
unanticipated  problems.  If  there  are  cost  and  schedule  constraints,  fully  document 
the  impact.  Include  risk  analysis  and  cost/benefit  analysis  to  prioritize  limited 
resources. 


Testing  is  often  hindered  beeause  of  inadequate  time 
available  for  testing . 


SCHEDULE  (40.5  PERCENT) 

This  group  of  findings  dealt  with  the  importance  of  an  adequate  schedule  and  the 
negative  effects  of  inadequate  schedule  on  T&E  programs.  Testing  is  often  hindered 
because  of  inadequate  time  available  for  testing.  Overly  optimistic  schedule  estimates 
commonly  lead  to  this  problem  as  well  as  unforeseen  problems,  which  decrease  the 
time  available  or  increase  the  time  needed  for  testing.  Program  managers  sometimes 
curtail  testing  in  order  to  make  up  for  lost  schedule  time.  Students  said  the  following: 

■  Take  the  time  to  understand  and  develop  a  realistic  plan  and  schedule,  including 
consulting  with  experts  as  necessary.  Plan  for  realistic  test  and  program  schedules, 
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with  time  allowed  for  things  such  as  maintenance,  bad  weather,  and  crew  rest.  Add 
cost  and  schedule  contingency  to  each  activity,  not  just  at  the  end. 

■  Communicate  the  need  for  adequate  scheduling,  including  probable  effects  if  needed 
or  planned  testing  is  delayed  or  cancelled.  A  limited  test  program  often  results  in 
higher  overall  program  costs  and  longer  overall  schedule  because  discovery  of 
problems  is  delayed.  If  there  are  schedule  constraints,  fully  document  the  impact. 
Include  risk  analysis  and  cost/benefit  analysis  to  prioritize  limited  resources. 


COMPARISON  WITH  OTHER  STUDIES 

A  literature  search  was  conducted,  and  the  results  from  this  study  were  compared 
with  the  results  from  five  other  similar  studies  and  documents.  These  studies  and 
documents  all  examined  and/or  presented  T&E  best  practices  and  lessons  learned.  A 
short  description  of  each  study  and  document  follows: 

DEFENSE  ACQUISITION  GUIDEBOOK  SECTION  9.8,  BEST  PRACTICES 
(DAG,  2004) 

This  list  of  T&E  best  practices  was  prepared  by  developmental  test  and  evaluation 
(DT&E),  operational  test  and  evaluation  (OT&E),  and  live  fire  test  and  evaluation 
(LFT&E)  experts  at  the  Office  of  the  Secretary  of  Defense  (OSD).  Best  practices  are 
offered  to  increase  the  likelihood  of  a  successful  T&E  program.  Some  commercial 
industry  T&E  best  practices  are  included  in  the  list.  However,  other  than  that,  the 
methodology  for  developing  the  DAG  list  of  best  practices  is  not  stated.  Since  the 
OSD  experts  (who  developed  DAG  Section  9.8)  have  extensive  knowledge  and 
experience  in  T&E  oversight  of  DoD  acquisition  programs,  presumably  the  list 
of  best  practices  is  (at  least  partially)  based  on  this  extensive  T&E  knowledge, 
experience,  and  expertise. 

A  MORE  CONSTRUCTIVE  TEST  APPROACH  IS  KEY  TO 
BETTER  WEAPON  SYSTEM  OUTCOMES  (GAO,  2000) 

In  this  report,  the  General  Accounting  Office  (GAO)  examines  (a)  how  the  conduct 
of  T&E  affects  commercial  and  DoD  program  outcomes,  (b)  how  best  commercial 
T&E  practices  compare  with  DoD’s,  and  (c)  what  factors  account  for  the  differences 
in  these  practices.  The  report  includes  detailed  discussion  of  DoD  and  commercial 
T&E  best  practices  and  lessons  learned,  along  with  recommendations  for  improving 
the  conduct  of  T&E  within  DoD.  The  GAO  conducted  literature  searches,  interviewed 
numerous  T&E  experts,  examined  four  DoD  weapon  programs,  and  analyzed  T&E 
best  practices  of  five  leading  commercial  firms,  including  site  visits  with  structured 
interview  questions  sent  in  advance  of  each  visit. 
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RECURRING  LESSONS  IN  WEAPON  T&E  PROGRAMS  (HOIVIK,  2000) 

This  article  summarizes  and  discusses  significant  issues  and  problem  areas  in 
conducting  DoD  T&E  programs.  More  detailed  analyses  and  findings  may  be  found 
in  two  Naval  Postgraduate  School  Master  of  Science  theses,  which  are  referenced  in 
the  article.  Sources  for  the  research  efforts  included  studies  of  T&E  in  various  major 
system  acquisition  programs,  including  information  from  program  office  personnel, 
testers,  analysts,  user  representatives,  and  contractor  T&E  personnel.  Literature 
searches  and  reviews  were  also  conducted. 

A  STUDY  OF  COMMERCIAL  INDUSTRY  BEST  PRACTICES  IN  TEST 
&  EVALUATION  WHICH  ARE  POTENTIALLY  APPLICABLE  TO  DOD 
DEVELOPMENTAL  TEST  AND  EVALUATION  (SAIC,  2002) 

This  study  presents  a  detailed  discussion  of  commercial  industry  best  practices, 
including  how  and  why  these  T&E  best  practices  have  led  to  industry  success. 

The  study  team  made  site  visits  to  12  leading  commercial  firms  and  asked  them  to 
identify  T&E  best  practices  that  make  them  successful  (structured  interview  questions 
were  sent  in  advance  of  each  visit).  The  team  met  with  senior  corporate  managers, 
engineers,  and  technicians.  The  four  focus  areas  for  gathering  information  were: 
philosophy,  policy,  and  approach;  test  investment;  test  execution;  and  test  evaluation. 

FLOYD  AND  WALLY'S  OPERATIONAL  TEST  AND  EVALUATION  TOP  10 
LESSONS  LEARNED  (SMITH  &  TUBELL,  2001) 

Using  knowledge  acquired  from  their  direct  and  indirect  experience  in  Army 
OT&E,  the  authors  share  and  discuss  their  hard-won  lessons  learned.  Their  top  10 
lessons  learned  are  presented,  along  with  advice  and  recommendations. 


COMPARISON  WITH  OTHER  STUDIES:  FINDINGS 

Table  3  compares  the  results  discussed  in  this  article  with  results  from  the  above 
five  studies  and  documents.  To  develop  Table  3,  it  was  determined  which  of  the  18 
lessons  leamed/best  practice  categories  were  listed  and/or  discussed  in  each  of  the 
five  documents.  The  18  categories,  ranked  by  frequency,  represent  only  the  results 
of  the  study  discussed  in  this  article.  The  author  did  not  attempt  to  rank  the  findings 
from  the  other  five  documents.  The  author  also  did  not  address  information  contained 
in  the  other  five  documents  that  was  outside  the  scope  of  the  18  categories. 

The  research  methodology  and/or  population  studied  were  different  in  each  of 
the  above  studies.  But  the  findings  were  similar  in  that  this  article’s  top  findings 
were  discussed  in  all  of  the  studies,  whereas  the  lower  categories  showed  up  in  less 
of  the  studies  (or  in  this  article’s  study  only).  It  is  significant  that  the  top  six  areas 
were  discussed  in  all  of  the  studies,  as  this  shows  that  the  same  sorts  of  issues  have 
repeatedly  surfaced  over  the  past  five  to  ten  years. 
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TABLE  3. 

TEST  AND  EVALUATION  BEST  PRACTICES/LESSONS  LEARNED 
STUDIES  AND  DOCUMENTS 


LESSONS  LEARNED/ 

BEST  PRACTICE 
CATEGORY 

MY 

FINDINGS 

DAG 

GAO 

HOIVIK 

SAIC 

SMITH 

AND 

TUBELL 

1 .  Test  design,  test  methods,  test 
execution,  and  analysis  methods 

X 

X 

X 

X 

X 

X 

2.  Test  planning 

X 

X 

X 

X 

X 

X 

3.  Teamwork  and  communication 

X 

X 

X 

X 

X 

X 

4.  Funding,  budget,  and  cost 

X 

X 

X 

X 

X 

X 

5.  Schedule 

X 

X 

X 

X 

X 

X 

6.  Test  infrastructure,  test  tools, 
test  articles,  and  instrumentation 

X 

X 

X 

X 

X 

X 

7.  Test  requirements 

X 

X 

X 

X 

8.  Safety  and  risk  management 

X 

X 

X 

X 

9.  Government  leadership/ 
management  issues,  and 
organizational  politics 

X 

X 

X 

X 

10.  Contractor  issues,  including 
contractor  leadership  and 
management  issues 

X 

X 

11.  Training  issues 

X 

X 

X 

12.  Modeling  and  simulation  (used 
in  conjunction  with  testing) 

X 

X 

X 

13.  Interfaces,  interoperability,  and 
integration  issues 

X 

X 

X 

14.  “Stuff  happens”  (Murphy’s  law, 
weather) 

X 

15.  Manpower  issues 

X 

X 

X 

16.  Immature  technology  and/or 
immature  system 

X 

X 

17.  COTS  and  Non-Development 
Item  (NDI)  issues 

X 

X 

18.  Poor  judgment 

X 
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CONCLUSIONS 

Despite  good  intentions  and  some  DoD  progress,  weapon  system  programs  still 
suffer  from  persistent  problems  associated  with  late  or  incomplete  testing  (GAO, 
2000).  Several  common  lessons  learned  have  surfaced  over  the  years,  as  evidenced  by 
the  similar  findings  from  this  study  and  the  five  other  studies  and  documents. 

The  purpose  of  this  article  is  to  identify  and  discuss  common  T&E  best  practices 
and  lessons  learned,  thereby  enabling  possible  cost  and  schedule  savings  and 
improved  test  results  from  future  T&E  efforts.  Testers,  evaluators,  and  program 
office  personnel  can  certainly  benefit  from  applying  these  lessons  where  they  are  not 
already  doing  so.  Decision  makers  and  acquisition/program  leadership  can  benefit 
by  better  understanding  the  top  T&E  related  issues,  as  reported  by  field-level  T&E 
personnel. 


Karen  M.  Stadler  has  1  7  years  of  engineering/acquisition  experience 
in  the  T&E;  systems  planning,  research  development,  and  engineering 
(SPRDE);  and  production,  quality,  and  manufacturing  (PQM)  career 
fields.  She  is  currently  a  professor  of  T&E  with  the  Defense  Acquisition 
University  (DAU),  and  teaches  all  DAU  T&E  classes  and  several  DAU 
systems  engineering  classes.  She  has  a  B.S.  in  chemical  engineering 
from  Purdue  University,  and  is  a  former  U.S.  Naval  officer. 

(E-mail  address:  karen.stadler@dau.mil) 
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IN  GENERAL 

We  encourage  prospective  authors  to  coauthor  with  others  to  add  depth  to  their 
submissions.  It  is  recommended  that  a  mentor  be  selected  who  has  published  before 
or  has  expertise  in  the  subject  presented  in  the  manuscript. 

Authors  should  become  familiar  with  the  construction  of  previous  Defense  ARJs 
and  adhere  to  the  use  of  endnotes  versus  footnotes,  formatting  of  bibliographies,  and 
the  use  of  designated  style  guides.  It  is  also  the  responsibility  of  the  corresponding 
author  to  furnish  government  agency/employer  clearance  with  each  submission. 


SUBMISSIONS 

We  welcome  submissions  from  anyone  involved  in  the  defense  acquisition 
process.  Defense  acquisition  is  defined  as  the  conceptualization,  initiation,  design, 
development,  testing,  contracting,  production,  deployment,  logistic  support, 
modification,  and  disposal  of  weapons  and  other  systems,  supplies,  or  services 
needed  by  the  Department  of  Defense  (DoD),  or  intended  for  use  to  support  military 
missions. 


RESEARCH  ARTICLES 

Manuscripts  should  reflect  research  or  empirically  supported  experience  in  one  or 
more  of  the  aforementioned  areas  of  acquisition.  Research,  lessons  learned,  or  tutorial 
articles  should  not  exceed  4,500  words.  Opinion  articles  should  be  limited  to  1,500 
words. 

Research  articles  are  characterized  by  a  systematic  inquiry  into  a  subject  to 
discover/revise  facts  or  theories. 
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MANUSCRIPT  SECTIONS 

A  brief  abstract  (120-word  limit)  provides  a  comprehensive  summary  of  the  article 
and  must  accompany  your  submission.  Abstracts  give  readers  the  opportunity  to 
quickly  review  an  article’s  content  and  also  allow  information  services  to  index  and 
retrieve  articles. 

The  introduction,  which  should  not  be  labeled,  opens  the  body  of  the  paper  and 
states  the  problem  being  studied  and  the  rationale  for  the  research  undertaken. 

The  methods  section  should  include  a  detailed  methodology  that  clearly  describes 
work  performed.  Although  it  is  appropriate  to  refer  to  previous  publications  in  this 
section,  the  author  should  provide  enough  information  so  that  the  experienced  reader 
need  not  read  earlier  works  to  gain  an  understanding  of  the  methodology. 

The  results  section  should  concisely  summarize  findings  of  the  research  and  follow 
the  train  of  thought  established  in  the  methods  section.  This  section  should  not  refer 
to  previous  publications,  but  should  be  devoted  solely  to  the  current  findings  of  the 
author. 

The  discussion  section  should  emphasize  the  major  findings  of  the  study  and  its 
significance.  Information  presented  in  the  aforementioned  sections  should  not  be 
repeated. 


RESEARCH  CONSIDERATIONS 

Contributors  should  also  consider  the  following  questions  in  reviewing  their 
research-based  articles  prior  to  submission: 

■  Is  the  research  question  significant? 

■  Are  research  instruments  reliable  and  valid? 

■  Are  outcomes  measured  in  a  way  clearly  related  to  the  variables  under  study? 

■  Does  the  research  design  fully  and  unambiguously  test  the  hypothesis? 

■  Are  needed  controls  built  into  the  study? 

Contributors  of  research-based  submissions  are  also  reminded  they  should  share 
any  materials  and  methodologies  necessary  to  verify  their  conclusions. 


CRITERIA  FOR  TUTORIALS 

Tutorials  should  provide  special  instruction  or  knowledge  relevant  to  an  area  of 
defense  acquisition  to  be  of  benefit  to  the  DoD  Acquisition,  Technology,  and  Logistics 
workforce. 

Topics  for  submission  should  rely  on  or  be  derived  from  observation  or 
experiment,  rather  than  theory.  The  submission  should  provide  knowledge  in  a 
particular  area  for  a  particular  purpose. 
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OPINION  CRITERIA 

Opinion  articles  should  reflect  judgments  based  on  the  special  knowledge  of  the 
expert  and  should  be  based  on  observable  phenomena  and  presented  in  a  factual 
manner;  that  is,  submissions  should  imply  detachment.  The  observation  and  judgment 
should  not  reflect  the  author’s  personal  feelings  or  thoughts.  Nevertheless,  an  opinion 
piece  should  clearly  express  a  fresh  point  of  view,  rather  than  negatively  criticize  the 
view  of  another  previous  author. 


MANUSCRIPT  STYLE 

We  will  require  you  to  recast  your  last  version  of  the  manuscript,  especially 
citations  (endnotes  instead  of  footnotes),  into  the  format  required  in  two  specific  style 
manuals.  The  ARJ  follows  the  author  (date)  form  of  citation.  We  expect  you  to  use  the 
Publication  Manual  of  the  American  Psychological  Association  (5th  Edition),  and  the 
Chicago  Manual  of  Style  (15th  Edition). 

Contributors  are  encouraged  to  seek  the  advice  of  a  reference  librarian  in 
completing  citations  of  government  documents  because  standard  formulas  of  citations 
may  provide  incomplete  information  in  reference  to  government  works.  Helpful 
guidance  is  also  available  in  Gamer,  D.  L.  and  Smith,  D.  H.,  1993,  The  Complete 
Guide  to  Citing  Government  Documents:  A  Manual  for  Writers  and  Librarians  (Rev. 
Ed.),  Bethesda,  MD:  Congressional  Information  Service,  Inc. 


COPYRIGHT  INFORMATION 

The  ARJ  is  a  publication  of  the  United  States  Government  and  as  such  is  not 
copyrighted.  Because  the  ARJ  is  posted  as  a  complete  document  on  the  DAU  home 
page,  we  will  not  accept  copyrighted  articles  that  require  special  posting  requirements 
or  restrictions.  If  we  do  publish  your  copyrighted  article,  we  will  print  only  the  usual 
caveats.  The  work  of  federal  employees  undertaken  as  part  of  their  official  duties  is 
not  subject  to  copyright  except  in  rare  cases. 

In  citing  the  work  of  others,  it  is  the  contributor’s  responsibility  to  obtain 
permission  from  a  copyright  holder  if  the  proposed  use  exceeds  the  fair  use 
provisions  of  the  law  (see  U.S.  Government  Printing  Office,  1994,  Circular  92: 
Copyright  Law  of  the  United  States  of  America,  p.  15,  Washington,  DC:  Author). 
Contributors  will  be  required  to  submit  a  copy  of  the  written  permission  to  the 
Managing  Editor  before  publication. 


COPYRIGHT  POLICY 

We  reserve  the  right  to  decline  any  article  that  falls  into  these  problem  copyright 
categories: 

■  The  author  cannot  obtain  official  permission  to  use  previously  copyrighted 
material  in  the  article. 
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■  The  author  will  not  allow  DAU  to  post  the  article  with  the  rest  of  the  ARJ  issue  on 
our  home  page. 

■  The  author  requires  that  unusual  copyright  notices  be  posted  with  the  article. 

■  To  publish  the  article  requires  copyright  payment  by  the  DAU  Press. 


MANUSCRIPT  FORMAT 

Pages  should  be  double-spaced  and  organized  in  the  following  order:  title  page, 
abstract,  body,  reference  list,  author’s  note  (if  any),  and  figures  or  tables.  Key  words 
should  also  be  provided  as  part  of  the  submission.  Figures  or  tables  should  not  be 
inserted  (or  embedded,  etc.)  into  the  text,  but  segregated  (one  to  a  page)  following  the 
text.  If  material  is  submitted  on  a  computer  diskette  or  e-mailed  each  figure  or  table 
should  be  saved  to  a  separate,  exportable  file  (i.e.,  a  readable  EPS  file).  For  additional 
information  on  the  preparation  of  figures  or  tables,  see  CBE  Scientific  Illustration 
Committee,  1988,  Illustrating  Science:  Standards  for  Publication,  Bethesda,  MD: 
Council  of  Biology  Editors,  Inc.  Please  restructure  briefing  charts  and  slides  to  a  look 
similar  to  those  in  previous  issues  of  the  ARJ. 

The  author  (or  corresponding  author  in  cases  of  multiple  authorship)  should 
attach  to  the  manuscript  a  signed  cover  letter  that  provides  all  of  the  authors’  names, 
mailing  and  email  addresses,  telephone  and  fax  numbers.  The  letter  should  verify 
that  the  submission  is  an  original  product  of  the  author;  that  it  has  not  been  published 
before;  and  that  it  is  not  under  consideration  by  another  publication.  Details  about 
the  manuscript  should  also  be  included  in  this  letter:  for  example,  title,  word  length, 
a  description  of  the  computer  application  programs,  and  file  names  used  on  enclosed 
diskettes  or  in  email  attachments,  etc. 


AUTHOR  PHOTOS 

Please  send  us  a  cover  letter;  biographical  sketch  for  each  author  (not  to  exceed  70 
words);  head  and  shoulder  print(s)  or  digitized  photo(s)  (saved  at  300  pixels  per  inch, 
at  least  5X7  inches,  and  as  a  TIFF  or  JPEG  file);  prints  of  photos  will  be  accepted 
and  returned  upon  request;  one  copy  of  the  printed  manuscript;  and  any  disks.  These 
items  should  be  sturdily  packaged  and  mailed  to:  Department  of  Defense,  Defense 
Acquisition  University,  Attn:  DAU  Press  {Defense  ARJ  Managing  Editor),  Suite  3, 
9820  Belvoir  Road,  Fort  Belvoir,  VA  22060-5565. 
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